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Message from the Program Chair 


Dear Reader: 


It took me a long time to find this conference. Even though I had already been 
working in the field for about 10 years, it was at my first LISA conference that I learned 
there was an entire world of system administration beyond my office — a world of 
research challenges, best practices, and a whole set of peers grappling with the same 
challenges. 


Now, many years after my first LISA, it is my pleasure as Program Chair to 
welcome you — or welcome you back — to this community. 


You have in your hand, on your screen (or in your implant, hello Future!) the full 
text of the papers accepted for presentation in the refereed papers track of the 19th LISA 
conference. It is one of the places where the history of our field is being written. 
Dedicated and talented people labored this year to create this body of work. Submission 
authors, program committee members, and external reviewers all devoted kerjillions of 
hours to the advancement of our field. I’m very grateful for their efforts and I know 
future generations of sysadmins will be as well. 


Similar valiant levels of effort have been expended by the Invited Talks 
coordinators (Adam and Bill), the coordinators for the various tracks and activities (Dan, 
Esther, Joe, Lorah, Luke, and Philip), and all of the other volunteers. Thanks are due to 
Rob who put the final polish on each paper until it gleamed bright enough to include in 
the collection in front of you now. I’ve also been very lucky to work with the fabulous 
USENIX staff, who help keep the train on the tracks no matter how the Program Chair 
drives. If you have appreciated this conference, please be sure to thank all of the LISA 
organizers/volunteers and the USENIX staff for their efforts. 


I have often thought that somebody should write an unabashed love letter to the 
sysadmin community. As I enter my twentieth year as a sysadmin, my work on this 
conference, the 19th LISA to date, is the closest I could come to such a thing. I hope this 
year’s LISA conference is as fabulous for you as my first LISA experience was for me. I 
look forward to joining you at many LISA conferences to come. 


Respectfully and with deep affection, 


David N. Blank-Edelman 
LISA °05 Program Chair 
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GULP: A Unified Logging 
Architecture for Authentication Data 


Matt Selsky and Daniel Medina — Columbia University 


ABSTRACT 


We have implemented the Grand Unified Logging Project, GULP, a flexible aggregation 
system for authentication log data. The system merges disparate logs stored across various servers 
into a single format according to an XML schema. This single format is logged to a database and 
queried via a web interface. The strength of this system lies in the ability to correlate information 
across multiple logging sources and display relevant information through a simple interface. 


Introduction 


At Columbia University, each person is given a 
unique username or “UNI” (University Network ID) 
by the Academic Information Systems (AcIS) group. 
A process is in place for UNIs to be activated, creating 
a password that allows access to various services. 


These services run on many different hosts and 
have disparate logging facilities. For example, when a 
student logs into CourseWorks (web-based course 
management), a successful authentication record is 
stored in the CourseWorks database. Other logins, to 
CubMail (web-based email), CUNIX and PINEX 
(shell servers), and elsewhere, are similarly logged, 
but to other locations on local disk [2]. 


Logging data is stored in a variety of formats and 
is typically stored locally on the host which provides 
the service. Some example services and the logs for- 
mats they use: 


UNIX wtmpx records (remote login to a server), 
unpacked from the binary format [5]: 


dnml7 pts/42 mutie.cc.columbia. edu 
Fri Oct 29 09:21 - 10:08 (00:47) 


Secure web servers run Apache and log in the 
common Apache text format [6]; see Display |. 


WIND (Web Identification Network Daemon) 
[3], which provides sign-on to various web applica- 
tions, has a custom text format: 
2004-10-29 09:21:00,000 Login - success 
for dnml7:switchmgr (128.59.31.101) 
[Paesisnstscel # 


CubMail is a webmail client based on the Horde’s 
IMP project [4]. Logs are text-based; see Display 2. 

We could send the logs via syslog to a remote host, 
but we currently have it configured to only log locally. 
Processing these logs is not as intensive as is seen in 
commercial enterprises, but our logs are still quite size- 
able [15]. During the summer months alone, there are 
about 10 GB/week of logs from the main webservers, 
and another 9 GB/week from the mail servers. Early in 
the school year we have observed 15.2 GB/week and 
12.5 GB/week for web and mail, respectively. 


Table 1 on the next page lists the main services 
we provide, along with the methods they use for log- 
ging authentication events. 


Problem 


Web and other logs are already harvested for 
usage statistics. Typically, there is an operational need 
to determine the number of users of a service, the appli- 
cations being used to access services (browser client, 
email client, etc.), and so on. These statistics are col- 
lected periodically, usually a few times each semester. 


The authentication information contained in these 
logs is valuable for a variety of other purposes. We gen- 
erate usage reports for budget requests and capacity 
planning, demographic reports to show which university 
divisions are using our services (and therefore which 
schools should purchase account upgrades for all their 
students), and client software reports to ascertain the 
software people are using to access our services. This 
helps us to determine utilization of site-licensed soft- 
ware and also to plan software to support in the future. 


mutie.cc.columbia.edu - dnml7 [29/0ct/2004:09:21:00 -0500] 
"(GET /sec/acis/networks/index.html HTTP/1.1)" 

200 202573 "(ref -)" " (client Mozilla/5.0 (Macintosh; U; PPC 
Mac OS X Mach-O; rv:1.7.3) Gecko/20040911 Firefox/0.10)" 


Display 1: Common Apache text format log entry. 
Oct 29 09:21:00 HORDE [notice] [imp] Login success for 


dnm17@columbia.edu [128.59.31.101] to {localhost:143} 
[on line 92 of "/etc/httpd/htdocs/horde/imp/redirect.php"] 


Display 2: A custom text-based log entry from CubMail. 
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GULP: A Unified Logging Architecture for Authentication Data 


Authentication information (UNI and password) 
gives access to an individual’s personal information, 
including payroll, financial aid data, grades and course 
registration, email, and personal contact information. 


We use Kerberos [1] to provide a centralized 
authentication mechanism, but Kerberos logs lack 
interesting features (since the transactions are bro- 
kered between the Kerberos server and the service or 
application server). We wish to preserve as much 
client information relevant to the authentication as 
possible. Several of our services do not natively sup- 
port Kerberos so they request a ticket on behalf of the 
user using the plaintext password. This type of event 
shows up in the Kerberos logs without any informa- 
tion about the end-host or any indication that the 
authentication attempt was successful; the log only 
shows that an initial ticket request was made from the 
service host for the user. 


The AcIS security group regularly receives 
requests regarding personal login information from 
persons who believe their accounts are compromised. 
A decade ago, before the explosion in web-based 
applications, it would suffice to direct users to the onall 
[7] command, which in conjunction with the last com- 
mand, would show the most recent logins on all of the 
UNIX timeshare hosts. last logs were never centralized 
as in [17], but the onall utility made searching for these 
standard logs easier. 


The security group also receives requests to deter- 
mine the owner of a particular host (usually in the form 
of an Internet Protocol (IP) address). Due to the model 
of local network access at Columbia (so-called ‘free 
love”), users are not required to log into the network to 
use it [8]. IP addresses may be linked to users via a 
multi-stage network logging procedure (mapping IP 
addresses to hardware addresses to switch ports to room 
information to room registration), or via authentication 
information. The latter is preferable when available. 


A recent problem for the security group has been 
various applications (sometimes called “SpyWare’’) that 
proxy user web traffic. Some of these applications proxy 
not just normal web traffic, but also SSL-protected 
(HTTPS) traffic.! This is equivalent to the user sending 
their UNI and password information to a third party 
(along with the data contained in the pages she is visiting). 


Since the Kerberos logs are not suitable for har- 
vesting the authentication data we want, we must look 


1By installing a trusted root certificate [9]. 
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to the logs stored on the individual servers, as 
described above. However these logs are not easily 
searchable through any standard interface. 


Too often, the AcIS security group finds itself in 
a reactive position, responding to incidents that have 
become operational threats (such as mass account 
compromise, break-ins, and malware epidemics). Dur- 
ing these incidents, there is typically a slow gathering 
of available information that may take hours and 
involve numerous staff members. Once logging infor- 
mation is centralized, we can gather this information 
much more rapidly. The application of data mining 
techniques may even enable proactive operations in 
the face of emerging threats. 


Solution 


We have implemented a flexible aggregation sys- 
tem that allows for easy querying of the relevant 
authentication data from disparate log sources. We 
gave the solution the moniker ‘““GULP” (Grand Uni- 
fied Logging Project). This was preferable to the half- 
serious “TIA” (Tracking ID Access/Total Information 
Awareness) and ““ECHELON” (Experimentally Cen- 
tralizing Host’s Every Log-On Name). We examined 
several mechanisms for extracting useful features 
from this data. 


Centralized Logging 


To centralize the authentication information we 
want, we transform the log files to an XML file that is 
described by an XML schema. This document includes 
only the “interesting” information that we have 
defined, reducing the total amount of data retained. 


Centralized Searching 


This single format is then searched for useful 
information via a web interface. The advantage of this 
system lies in the ability to correlate information across 
multiple logging sources easily. Advanced searches 
can be defined and saved for future re-evaluation. 


Data Mining 


With the limited features we have from our authen- 
tication data, we can extract information regarding 
abnormal behavior. For example, evidence of spyware or 
a proxy server would be many different users connecting 
from a single source or network (off the campus net- 
work). A login from a source not seen before may indi- 
cate unauthorized access to an account. The ultimate 
goal, however, would be to provide a system to allow a 


Service Function Logging 
CUNIX Shell servers (General purpose) local, wtmpx 
PINEX Shell-based E-mail local, wtmpx 
CubMail Web-based E-mail local, custom 
CourseWorks Course-related materials remote DB 
Secure Web SSL-protected pages on www! local, Apache 


WIND Web-app sign-on platform 
VPN and dialup authentication 


RADIUS 


local, custom 
local, RADIUS detail 


Table 1: Main services provided. 
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member of the security group to create rules as needed, 
rather than using hard-coded signatures. 


Implementation 


Centralized Logging 


The advantage to using XML is that we can pub- 
lish our schema and leave the responsibility for 
extracting the relevant data from the logs to the main- 
tainer of the application generating the logs. The 
maintainer can then validate the generated document 
before contributing it to the central repository.2 While 
we did not interact with any outside parties for the 
purposes of this project, it is easy to perform the 
required validation. The schema used is included in 
the Appendix. 


The XML representations of the log files may 
then be transferred and stored in a relational database 
(MySQL, in our case) [10]. 


For our project, we used logs from UNIX time- 
share hosts (CUNIX and PINEX), web application 
log-in servers (WIND), secure web servers (WWWS), 
and webmail (CubMail), all of which we have 
described above. We chose the timeshare hosts 
because “last” logs are traditionally important sources 
of remote login information (although fewer users 
now log in directly). We chose WIND logs because it 
controls access to some of the most important web- 
based applications at the university, including payroll 
information. We chose secure web logs because the 
format is very common and numerous proxies can be 
found and accessed via the secure web servers. We 
chose CubMail because it is a popular application, 
used by approximately 60% of our users. 


We created simple parsers for each of the logs we 
intended to use. In the case of the “last”, Apache, and 
Horde log parsers, we hope to have decent reuse poten- 
tial, while WIND logs will probably be unique to our site. 


Writing the parsers is fairly simple, with most 
parsers being less than 150 lines of code (and much of 
the code just setting up the connection to the data- 
base). The difficult parts of the parser are writing the 
regular expressions to extract revelant data from each 
log entry in the log file, dealing with disparate date 
and time formats, and reading binary log data. The 
more complex parsers also need to attempt to re-create 
session information using login and logout records 
that may not match up. We do not currently attempt to 
canonicalize usernames to the UNI since we do this in 
the web interface, by searching for the UNI and any 
other usernames associated with a person. 


Centralized Searching 
We created a simple web form, protected by an 
-htaccess file restricting access to our Security group, 


2Experience has shown that publishing a required schema 
P : ; p g q 1 
that cannot be easily validated by the source and repository 
parties is pointless. 
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that allows searching via a username or remote IP 
address. The information returned includes the remote 
hostname, the service used, the local server that 
exported the log, the start and end time of the session 
(only the start time where the session concept does not 
apply), and a note if applicable (such as the TTY used, 
or the web page or service accessed). A link to an 
external WHOIS site is included for more information 
about the remote host [11]. 


The username and remote address are also linked 
back to the CGI to allow easy inversion of the search 
on either term. This type of inversion is quite typical 
(when, say, trying to determine what other logins have 
come from a strange address). In this manner, we have 
slightly more features than a simple log-grepper allows. 


We have also created an additional web form to 
show users their own authentication history, after log- 
ging in, but we have not deployed it as of yet. 


Data Mining 


We had several pre-conceptions about what 
would constitute an anomalous login. We conjectured 
that there would be two kinds of (global) abnormali- 
ties: many connections coming from a single address 
for many different users; and a single user logging in 
from many different locations. 


We collected frequency statistics for both of the 
above abnormalities and quickly discovered that our 
assumptions were not refined enough. Of 40,000 user 
accounts observed over three weeks, over 10,000 were 
seen from more than six remote locations. Almost 
6,000 were seen logging in from over 10 locations. (10 
users from a German ISP logged in from more than 60 
distinct locations within that ISP’s address space). 


We attempted to use BGP [12] information to 
retrieve the Autonomous System (AS) number associ- 
ated with each remote IP address. In this case, a user’s 
60 remote locations would be represented by a single 
AS number (belonging to the ISP). According to these 
new metrics, we find that most users typically log in 
from fewer than three ASs (work, home, and the 
Columbia network). Nevertheless, with 40,000 users, 
we will still experience many false alerts (and miss 
many legitimate violations). 


In the other direction, we found almost 900 
addresses (of over 120,000) from which more than 12 
users logged into our systems. Of these, we know that 
some are classified by our security group as malicious 
proxies (as described above). Many of the remaining 
addresses belong to benign web proxies, NAT-ing 
routers, and corporate gateways. 


Applications 


In the end, the search tool is more useful for 
revealing anomalous behavior than a global set of 
rules. Allowing limited access and providing a useful 
interface, we can deploy the search tool to the larger 
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user community. Colorizing logins from various 
sources by network, a user may easily audit her own 
login history [13, 16]. 


A user is more likely to be aware of what quali- 
fies as an anomalous login than a system administrator 
responsible for thousands of users (especially given 
our diverse user population and the limit of the log 
records we are processing). Making this record avail- 
able to the user is no different than a phone bill or 
credit card bill, an itemized list that the user can use to 
check for fraudulent activity and transactions. We 
show an example of this report in Figure | below. 


We used this user search tool to investigate twelve 
recent security incidents reported by end-users. One 
such incident involved a student travelling overseas; he 
had used a computer terminal “‘administered by a guy 
who admitted to me in a moment of intoxication that 
he’s a criminal hacker.” Needless to say, the student 
was concerned about the security of his account. 


In six of the twelve cases, the tool confirmed the 
suspicions of the user that someone else was using 
their account. In the other cases, we did not observe 
any anomalous patterns, possibly due to either gaps in 
our data or gaps in our coverage (we are not yet col- 
lecting data from all available log sources). In one 
instance we identified a supposedly secure web appli- 
cation on a departmental server that was in fact using 
plaintext ftp for file uploads to CUNIX. 


In another incident, the identity of the miscreant 
was discovered. A student suspected that someone 
was reading her email because she often found her 
message flags altered. Using the search tool, the secu- 
rity group found a number of abnormal logins from a 
public campus terminal. Inverting the search on the 


{medina 160.39.246.251 
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public terminal, they found that the same individual 
had logged in to the terminal before the complaining 
student. Apparently, the miscreant would check his 
mail first, then hers. 


The security group can also create custom 
searches as required. The search shown in Figure 2 
below quickly identifies all malicious MarketScore 
proxies (as defined above). When these proxies were 
first identified, it took two days to formulate the entire 
list of proxy sources and countless staff time since dif- 
ferent staff members were familiar with the log loca- 
tions and contents for different services. Currently, 
logs are collected of users of the proxies once a week. 
With this tool, a current list can be created instantly. 


Future Work 


Numerous areas for development are open to us 
now that we have a viable central logging system. We 
also see a number of improvements that can be made 
to the applications we have already created. 


We will take steps to properly normalize the 
username (certain logs do not record the UNI of an 
individual and instead log a username, which in the 
case of staff members, may not be equivalent). We 
have currently handled this in the web form. 


We will further improve the idea of a “session” 
by correlating login and logout messages from some 
of the sources that did not clearly identify records as 
belonging to a particular session (as has been done 
with CubMail logins). 


We will expand the logs that we feed into the 
system, including POP, IMAP, authenticated SMTP, 
RADIUS, and CourseWorks logs. 


2004-11-29 17:11:25 [2004-11-29 17:22:19 


11-29 18:09:58 [2004-11-30 01:24:30 
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Figure 1: Sample search on user medina. 
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Figure 2: Sample MarketScore logins (users obfuscated). 
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We will improve the user-facing application to 
query personal login information. Any user tools that 
decrease support staff time are a boon. 


We will further evaluate possible machine learn- 
ing algorithms with more satisfactory error rates, and 
possibly incorporate these algorithms into the user- 
facing tool. 


We will look at using real-time log-processing 
frameworks, such as SHARP, to collect information as 
it is available [14]. 

We will further research more widely-used stan- 
dards for sharing logging messages, such as Inter- 
net2’s ccBAY and Conostix’s IPFC [18, 19]. 


Availability 


This paper and related code can be found online 
at http://www.columbia.edu/acis/networks/advanced/ 


gulp/. 
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Appendix 
XML Schema 


<?xml version="1.0"2> 

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
xmlns: gulp="http://www.columbia.edu/xml/gulp" 
targetNamespace="http://www.columbia.edu/xml1/gulp" 
elementFormDefault="qualified"> 


<xsd:annotation> 
<xsd:documentation xml:lang="en"> 
[G]rand [U]nified [Llogging [P]roject schema for AcIS 
</xsd:documentation> 
</xsd:annotation> 


<xsd:simpleType name="UNI"> 
<xsd:restriction base="xsd:string"> 
<xsd:pattern value="==>[ignored: w]<==+"/> 
</xsd:restriction> 
</xsd:simpleType> 


<xsd:simpleType name="timestamp"> 
<xsd:restriction base="xsd:string"> 
<xsd:pattern value="(\d{4}\d-\d{2}-\d(2}\ds\d(2}:\d(2}:\d{2})2" /> 
</xsd:restriction> 
</xsd:simpleType> 


<xsd:simpleType name="ip_addr"> 
<xsd:restriction base="xsd:string"> 

<xsd:pattern value="[{\d\.]+" /> 
</xsd:restriction> 
</xsd:simpleType> 


<xsd:complexType name="bLog"> 


<xsd:all> 
<xsd:element name="uni" type="gulp:UNI"/> 
<xsd:element name="starttime" type="gulp:timestamp"/> 
<xsd:element name="endtime" type="gulp:timestamp"/> 
<xsd:element name="service" type="xsd:string"/> 
<xsd:element name="server" type="xsd:string"/> 
<xsd:element name="remote_addr" type="gulp:ip_addr"/> 
<xsd:element name="remote_hname" type="xsd:string"/> 
<xsd:element name="note" type="xsd:string"/> 

</xsd:all> 


</xsd:complexType> 


<xsd:element name="logs"> 
<xsd:complexType> 
<xsd:sequence> 
<xsd:element name="basicLog" type="gulp:bLog" 
maxOccurs="unbounded"/> 
</xsd:sequence> 
</xsd:complexType> 
</xsd:element> 


</xsd:schema> 


Database Schema 


create table basic_log ( 

uni varchar(8) not null, 

starttime datetime, 

endtime datetime, 

-~ well-defined service name 

service varchar(25) not null, 
server node name 
server varchar(20) not null, 
remote_addr int unsigned not null, 
remote_hname varchar(75), 
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Is 


create 
create 
create 
create 
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note 


varchar(200), 


id int not null auto_increment, 
primary key (id) 


index 
index 
index 
index 


basic_log_uni on basic_log(uni) ; 
basic_log_remote_hname on basic_log(remote_hname) ; 
basic_log_remote_addr on basic_log(remote_addr) ; 
basic_log_ service on basic_log(service) ; 





~ 





Toward an Automated Vulnerability 
Comparison of Open Source IMAP Servers 


Chaos Golubitsky — Carnegie Mellon University 


ABSTRACT 


The attack surface concept provides a means of discussing the susceptibility of software to 
as-yet-unknown attacks. A system’s attack surface encompasses the methods the system makes 
available to an attacker, and the system resources which can be used to further an attack. A 
measurement of the size of the attack surface could be used to compare the security of multiple 
systems which perform the same function. 


The Internet Message Access Protocol (IMAP) has been in existence for over a decade. 
Relative to HTTP or SMTP, IMAP is a niche protocol, but IMAP servers are widely deployed 
nonetheless. There are three popular open source UNIX IMAP servers —- UW-IMAP, Cyrus, and 
Courier-IMAP — and there has not been a formal security comparison between them. 


In this paper, I use attack surfaces to compare the relative security risks posed by these three 
products. I undertake this evaluation in service of two complementary goals: to provide an honest 
examination of the security postures and risks of the three servers, and to advance the study of 
attack surfaces by performing an automated attack surface measurement using a methodology 
based on counting entry and exit points in the code. 


Introduction 


System administrators frequently confront the 
problem of selecting a software package to perform a 
desired function. Many considerations affect this deci- 
sion, including functionality, ease of installation, soft- 
ware support, interoperability, performance, and hard- 
ware and software dependencies. However, the sensi- 
ble administrator will place significant weight on 
choosing a software package with a reasonable posture 
towards security. 

When selecting a package and installing it ini- 
tially, the administrator can ensure relatively easily that 
the new configuration is not susceptible to any known 
vulnerabilities. However, some software is more prone 
to future attacks than others. Sysadmins could benefit 
from an easy-to-implement metric which provides an 
intelligent assessment of which software packages are 
likely to be found vulnerable in the future, not just of 
which ones have had problems in the past. 


Measurement of the attack surface of a software 
package has been proposed as such a metric. The 
attack surface is the set of opportunities which the 
software package provides for the outside world to 
access the software itself. It is measured by enumerat- 
ing this set and classifying its elements on the basis of 
risk of future attack. Researchers have further pro- 
posed automating attack surface measurement by enu- 
merating and classifying the entry and exit points of 
the software package. In this paper, I describe steps 
taken towards performing an automated relative vul- 
nerability assessment of open source UNIX IMAP 
servers based on measurement of attack surfaces. 


Contributions and Roadmap 


The paper makes two major contributions. First, 
I undertake an in-depth discussion of the relative secu- 
rity postures of the three major open source IMAP 
servers in use today. Second, I perform a partial mea- 
surement of attack surfaces based on the entry/exit 
point methodology, and compare the results to my ini- 
tial impressions of the servers studied. 


In the remainder of this section, I introduce attack 
surfaces and IMAP servers. I discuss the prior work 
done on attack surface measurement and the rationale 
for selecting IMAP servers as the target of my analy- 
sis. I discuss IMAP servers at the design level by first 
exploring the high-level interactions of the servers 
with their environments, then introducing the three tar- 
get servers with a focus on the design philosophies and 
implementation peculiarities of each. In the next sec- 
tion, I introduce the entry/exit point method of attack 
surface measurement, and discuss the methodology 
used in performing a partial entry/exit analysis of the 
IMAP servers, including roadblocks to performance of 
a fully automated analysis. Finally, I compare the 
results of the analyses performed, and provide conclu- 
sions and future directions for attack surface research. 


Attack Surfaces 
Theory of Attack Surfaces 

The attack surface can be used to create a com- 
parative software vulnerability metric for packages 
which perform similar functions. Such a_ metric 
assesses a system at the design level by observing the 
prerequisites for a system to be attacked. 


An attack on a system necessarily involves an 
attacker who gains access to some resource to which 
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he is not entitled. There are some prerequisites without 
which an attack cannot occur. First, the attacker must 
be able to take some action which affects the system. 
If he cannot alter a system’s state at all, he cannot 
increase his access to that system. Second, in order for 
an attack to be non-trivial, the system must contain 
some resource which is accessible after the attack, but 
not before. This resource may itself be the target of the 
attack, or it may be an enabler which allows the 
attacker to reach his eventual target, or to get closer to 
it. For example, in the case in which an unauthenti- 
cated attacker triggers a buffer overflow in a port-lis- 
tening daemon to obtain a root shell on the server, the 
action is the daemon, which runs code based on con- 
nections received from anyone, and the resource is the 
daemon’s root privilege. 


The attack surface of a system is simply the set 
of all actions made available by the system and all 
resources accessible to the system. This set consists of 
three types of items: (1) methods, which are exe- 
cutable code potentially runnable by an attacker, (2) 
channels, which are IPC mechanisms potentially 
usable by an attacker, and (3) data items, which are 
sources of persistent data (such as files) potentially 
readable or writable by an attacker. 


However, enumerating system actions and 
resources is not very useful in itself, because it tells us 
nothing about the specific risks posed by the actions 
and the specific opportunities afforded by the 
resources. A port-listening daemon which requires 
connections to come from a specific IP address and 
which asks clients to cryptographically authenticate 
themselves before communicating further may pose 
less risk than does a daemon which will allow any 
client to talk to it. We need some way to discuss the 
relative risks posed by different methods or resources. 


Prior Work on Attack Surface Measurement 


The idea of attack surfaces was introduced by 
Howard, and was used to measure the relative attack 
surfaces of versions of Windows [11]. Manadhata and 
Wing extended the description of attack surfaces in 
2004, and discussed ways of measuring attack sur- 
faces in the context of Linux versions [14]. Their 
approach to the problem that not all system resources 
are created equal was to use the Common Vulnerabili- 
ties and Exposures (CVE) database [1] to identify sev- 
eral types of objects which were most often involved 
in attacks on Linux. Typed objects include things like 
“http daemon running as root” or “file writable by 
group users.” They then enumerated those types 
within instances of each of several Linux distributions 
to draw a comparison. 


Several improvements on their analysis are envi- 
sioned here. First, manual analysis is tedious and 
error-prone, so a means of automating attack surface 
measurement is desired. Second, it is not possible to 
use this method to compare attack surfaces between 
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two products — if one has more daemons running as 
root, but the other has more world-writable files, we 
cannot say anything further about which poses the 
greater risk. Third, identifying important resources 
based on previously discovered vulnerabilities assumes 
both that the set of known vulnerabilities is large 
enough to be representative’ and that future attacks 
will exploit the same types of resources and actions as 
previous attacks. In order to focus on features which 
could be attacked in the future, we need to define 
types in a way which more accurately assesses the risk 
introduced by each. The entry/exit point analysis dis- 
cussed in the third section of this paper attempts to 
address these issues. 


IMAP Servers 


The Internet Message Access Protocol provides 
e-mail access to authenticated remote users. Once a 
Mail Transfer Agent receives a message for a local 
recipient, it must place that message into a data store 
from which the recipient can access it at his leisure. In 
many cases, the MTA places the message directly into 
a file, using a mail storage format such as Maildir or 
Berkeley mbox. The recipient then logs onto a UNIX 
system which has access to the mail file, and uses a 
UNIX-based mail client (MUA) to read the message. 


IMAP provides a mechanism for the mail store 
to be accessed over a network. The MTA hands a mes- 
sage off to the IMAP server, either by storing it in a 
file format the server can read, or by communicating 
with the IMAP server using a mail transmission proto- 
col such as SMTP or LMTP (Local Mail Transfer Pro- 
tocol). At a later time, the recipient may connect to the 
IMAP server over a network, authenticate, and per- 
form actions such as reading new messages, deleting 
messages, or selecting messages from his mailbox 
based on certain criteria [15]. 


I chose to investigate IMAP for several reasons. 
First, there are three freely-available open source IMAP 
servers which share the bulk of the market - UW-IMAP., 
Cyrus, and Courier-[MAP. Each server has been in devel- 
opment for several years, so the codebases are fairly 
mature. Each codebase contains 100,000-250,000 lines of 
code. These codebases are large enough that it is not 
practical to perform an analysis by reading every line of 
the code, but small enough that it is possible to get a 
design-level sense of the actions the code takes. Second, 
since IMAP is a niche protocol relative to, for instance, 
SMTP or HTTP, there has not been a formal security 
analysis of IMAP servers in the past. Third, the IMAP 
protocol is stateful, and, in particular, provides for both 
authenticated and unauthenticated states. In theory, attack 
surfaces are well suited for analysing stateful protocols, 
because they take into account what privileges are prereq- 
uisites for attacking a certain portion of a system. For all 


these reasons, studying the attack surfaces of IMAP 


‘This is certainly not the case for IMAP servers, for which 
only about 30 distinct server vulnerabilities are reported in 
the CVE, 
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servers is both feasible and worthwhile, in that it provides 
useful information about server security. 


Observation of IMAP Server Software 


To assess the effectiveness of the attack surface 
analysis, I needed to gain a subjective impression of 
the relative security postures of the three IMAP server 
packages. In order both to increase my knowledge of 
IMAP server operation, and to build a consistent base- 
line from which to compare the codebases, I installed 
each server package in a constrained environment. To 
the extent possible, I applied the same minimal config- 
uration to each server. 


Observation Methodology 


I installed each package using the jail() system 
call under FreeBSD 5.2. The jail() call is an extension 
of chroot() which restricts a called program and its chil- 
dren to operating within a subdirectory of the filesys- 
tem. All dependencies of the jailed program, including 
libraries, configuration files, devices, and log files and 
sockets, must exist within that subdirectory. In addi- 
tion, jail() binds all network activity of the jailed 
process and its children to one given IP address [12]. 


Using jail(), I was able to create a miniature vir- 
tual server for each IMAP package. Each virtual 
server ran only three items: a syslog daemon provid- 
ing debug-level system logging for all activity, a lis- 
tener of some sort to receive e-mail via LMTP connec- 
tions, and the IMAP daemon itself. In addition, I cre- 
ated a virtual server containing the Postfix MTA, 
whose purpose was to forward mail via LMTP to each 
of the three IMAP servers. 


For each IMAP server, I attempted to create a 
minimal default configuration in which the server lis- 
tened only on port 143 for both encrypted and unen- 
crypted connections.? Each server provided access to 
one user account, which could be authenticated via a 
CRAM-MDS database. 


I determined the minimal set of files which each 
server needed in order to function by trial and error. I 
modified the jail’s startup routine to start all programs 
as children of ktrace(), and used that in combination 
with syslog error output to iteratively determine what 
additional files were needed. Needless to say, this pro- 
cedure does not give an entirely comprehensive view of 
all the files a given program might use under any cir- 
cumstances. However, it does enable some subjective 
assessment of the behavior of each package, including: 
installed size and dependencies, behavior during com- 
pilation and configuration, network and system behav- 
ior during operation, and any notable extra features the 
software provided, be these user-visible options, extra 
daemons and port listeners, or API methods. 


2Technically, the port listens only for unencrypted connec- 
tions. However, it is possible for a client to connect and im- 
mediately issue the STARTTLS command, thereby encrypting 
the remainder of the session. 


Toward an Automated Vulnerability Comparison of Open Source IMAP Servers 


In the remainder of this section, I will introduce 
each of the three IMAP servers in turn, discussing the 
project overview of the package, the code layout and 
high-level design, my experience in installing and 
configuring the package, and my subjective impres- 
sion of the security of the package. 

High-Level Interactions of IMAP Servers 


Some high-level elements are common to all 
IMAP servers, as a result of the services they provide. 
In particular, there are several ways in which an IMAP 
server must interact with its environment in order to 
do its job. These interactions correspond to types of 
attack surface elements which all IMAP servers must 
have, so it is worthwhile to outline them in a general 
sense before proceeding to the specifics of each server 
package. Figure 1 depicts the interactions described in 
the remainder of this section. 


OS 
Programs 


server machine 












Figure 1: High-level IMAP server interactions and 
dependencies. (Executables are white, files are 
grey. Imapd may provide Imtpd service inter- 
nally). 


Remote API 


First and foremost, the IMAP server provides a 
network API (application program interface) to a 
remote user. The server listens on a port — commonly 
port 143 for unencrypted or encrypted connections, or 
port 993 for encrypted connections only — and 
responds to user connection requests. 


The IMAP API is the set of commands which the 
IMAP server considers to be legal for the remote client 
to send. It consists of 30-50 commands, the majority 
of which are specified by the IMAP RFC. The user 
sends these commands to the server using a fixed for- 
mat and a command-specific number of arguments, 
and each command has an RFC-defined effect. The 
command set includes actions such as CAPABILITY (list 
the features offered by this IMAP server), STARTTLS 
(negotiate encryption for this connection), AUTHENTI- 
CATE (begin some form of negotiation to move from 
an unauthenticated state to an authenticated state), and 
SELECT (choose a mailbox for further operations). 


Most of the commands deal directly with pro- 
cessing of e-mail, and thus should be accessible only 
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to authenticated users. However, a set of authentica- 
tion commands must be available to anyone who con- 
nects to the server, so that authentication can take 
place. In addition, some servers offer support for fold- 
ers available to anonymous users, and thus for an 
anonymous login mechanism.? 


Access to User E-Mail Data 


In order to serve e-mail, the IMAP server must 
have a way of receiving that mail from the MTA. The 
e-mail reception mechanism can take one of two forms: 
either the MTA deposits the mail in a format the IMAP 
server can read, or the IMAP server listens over a net- 
work or named socket for connections from the MTA. 
In the former case, the MTA and the IMAP server must 
share disk access. In the latter case, the IMAP server 
must run another listening daemon which speaks a mail 
transfer protocol such as SMTP or LMTP. 


Once the mail has been received, the IMAP 
server must be able to read and modify the messages 
on disk, and must maintain e-mail metadata in order to 
keep track of mailbox state required by the IMAP pro- 
tocol. Typically, metadata is stored as separate files 
associated with each user or each mailbox, or as 
header data within mail files. 


Operating System Dependencies 


The IMAP server depends on its resident operat- 
ing system (a version of UNIX in the cases of my 
three reference servers) to provide it with several nec- 
essary functions. First, there is a standard set of utili- 
ties and system libraries — such as Is and libe — which 
are needed by almost any running program. 


Second, the IMAP server needs a way to listen 
for incoming connections over the network. Some 
servers implement their own networking code, while 
some take advantage of OS-provided daemons such as 
inetd to launch the IMAP server in response to connec- 
tion requests. 


Third, each IMAP server uses some third-party 
libraries to provide various functions. All three servers 
make use of OpenSSL to encrypt the communication 
channel with the remote user. In addition, some of the 
servers use libraries such as BerkeleyDB to process 
authentication or metadata files in database formats. 


Access to Non-Mail Files 


Lastly, each server makes use of a number of files 
which are not directly accessible to the remote user or 
related to e-mail processing, but are used by the server 
backend. These files are of four primary types: (1) con- 
figuration files to be read at server startup or during 
operation, (2) log files or a logging subsystem to be 
written with error and/or debugging output, (3) authenti- 
cation files or an authentication subsystem to be con- 
sulted when a remote user attempts to prove his identity, 


3This mechanism typically allows for login with the user- 
name anonymous and any valid text string as a password, and 
should be familiar to users of FTP. 
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and (4) lock files and temporary files to be written in 
order to store non-permanent data or to control access to 
shared elements. 


UW-IMAP Server 
Overview 


UW-IMAP [9] is written and maintained at the 
University of Washington by Mark Crispin, the author 
of the original IMAP RFC. The purpose of this package 
is to provide a simple and flexible drop-in IMAP server 
for multi-user systems. The package uses the assump- 
tion that IMAP will be one of many login methods 
through which remote users can access the system. In 
particular, the functional differences between IMAP 
access and a shell access method such as SSH should be 
only that IMAP access is optimized for mail reading. 
Restricting IMAP access beyond the access afforded to 
a shell user is not a design goal. 


The UW-IMAP server has been under active devel- 
opment since 1988, though the entire codebase has been 
rewritten several times since then. The current code is 
considered to go back only as far as the 2000 imap-2000 
release. Looking further back, I find a code overlap of 
approximately 20% between imap-2004c1 (the most 
recent version as of this writing) and the 1996 imap-4 
release, and no overlap between imap-2004c1 and any 
release prior to imap-4. 


The current codebase contains 135,000 lines of 
code and 40,000 lines of other files. Of this code, the 
IMAP server itself comprises only 4,000 lines, while 
the remainder of the code consists of an internal (com- 
piled-in) library called c-client. This library is also the 
backend for the Pine e-mail client. 

Compiling imapd provides a single binary with a 
single purpose. An external program such as inetd 
must be used to listen on the appropriate IMAP ports. 
When a connection is made, an imapd process is 
spawned, handles that single connection, then termi- 
nates. Since UW imapd’s place in the system is simple, 
the amount of code needed for its implementation is 
reduced. The tradeoff is increased dependencies on 
other programs to perform core functions, most 
notably mail delivery and port listening. The imapd 
program also requires no configuration file — configu- 
ration options are to be selected at compile time. 


One more notable feature of UW-IMAP is that it 
is agnostic about mailbox formats. By default, the 
UNIX UW installation is compiled with support for 
mbox, mbx, mx, mh, tenex, mtx, mmdf, and phile 
mailbox types. This support is provided by means of 
mailbox drivers. Internal logic is used to guess the 
type of a mailbox, and then execution is passed off to 
the appropriate driver. 


Impressions 


The UW codebase does not have a good security 
history, and the design does not inspire confidence about 
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its future. My survey of IMAP vulnerabilities found in 
the CVE database (through December 2004) found 
twelve relevant vulnerabilities in the UW server, com- 
pared to seven in Cyrus and four in Courier. UW- 
IMAP’s primary benefit is that it is the smallest and 
simplest of the three servers, both in terms of code size 
and major functions provided, and in that it provides a 
smaller set of IMAP API methods than the other 
servers. (The small API set may be in part due to the 
fact that the UW author wrote the IMAP RFC, which 
defines the minimal allowable set of API functions.) 


However, the drawbacks are many, and seem to 
go down to the design philosophy of the package. The 
code is not at all modular. An example of this is given 
by the imapd program’s main() routine. The routine 
contains code to select a mailbox sorting function for 
the SORT API method. It also contains three separate 
code locations at which the anonymous user is config- 
ured (a pre-authentication check at the beginning of 
the connection, an outcome of the LOGIN API method, 
and an outcome of the AUTHENTICATE API method), 
causing three distinct variables to be set each time. In 
addition, since the codebase is so non-modular, and 
since most of the functionality is provided by a c- 
client library which is also the backend for the mail 
client Pine, it is possible that functionality may be 
compiled in to the UW server which is really only 
necessary or desirable for client operation. 


The design decision that IMAP access should be 
only one method into a user-accessible system effec- 
tively prevents administrators from building a closed 
box IMAP server. (UW can be compiled in a mode 
called closedBox, but the differences from the standard 
configuration are limited, and this is not an officially 
supported configuration.) The assumption that the 
server will not be a closed box shows up in several 
places. For example, one of the methods by which the 
Pine client attempts to contact an IMAP server uses 
rsh or ssh to run a server instance locally. In order to 
support this, the imapd server offers a ‘“‘pre-authenti- 
cated” mode, in which a server started under a given 
UID is assumed to be authenticated as the associated 
IMAP user. 


Despite UW-IMAP’s history of buffer overflows, 
instances of string functions which do not perform 
length-checking (such as sprintf) are still plentiful 
within the code. The codebase uses custom makefiles, 
rather than GNU configure or some other standard 
solution, to maintain the cross-platform build process. 
This choice is not necessarily related to security, but it 
causes code compilation to be less uniform than might 
otherwise be the case. Along the lines of the closedBox 
option, the package officially disallows use of a con- 
figuration file, but the code specifies the name of a 
configuration file which will be read if it exists. 
According to the documentation, the results of using 
this file to configure the software are unpredictable. 
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Cyrus Server 
Overview 


Cyrus is written and maintained by Project Cyrus 
at Carnegie Mellon University [4]. Its purpose is to 
provide fast and scalable IMAP service. In order to 
support this goal effectively, the Cyrus server is, by 
design, truly a closed box. Cyrus permissions are 
defined internally and do not map to the UNIX per- 
missions of the host operating system, and Cyrus uses 
its own custom mailbox format which is not guaran- 
teed to be parsable by non-Cyrus tools. 


Cyrus has been under active development since 
1994. In approximately 1999, the codebase was split 
into cyrus-imap, the IMAP server, and cyrus-sasl, a set 
of flexible authentication libraries and associated utili- 
ties for use with IMAP or other Cyrus programs. 


The Cyrus codebase contains 210,000 lines of 
code and 475,000 lines of other files. It is therefore the 
bulkiest of the three codebases, but is also relatively 
well documented. Of the included code, 75,000 lines 
come from the SASL codebase (the wrapper authenti- 
cation libraries themselves, the optional authentication 
daemon, plugins for common authentication mecha- 
nisms, and utilities for checking and changing pass- 
words), while the remainder is the IMAP codebase. 
The IMAP side of the code provides a number of aux- 
iliary tools and functions, but the code specific to 
imapd itself is nearly 70,000 lines. 


Compiling IMAP and SASL produces a large 
number of binaries and libraries, many of them 
optional. In order to function at all, the system 
requires: the front-end daemon master to listen for 
incoming network connections and pass them off to 
the appropriate servers, the imapd binary to handle 
IMAP connections, the Imtpd binary to receive incom- 
ing mail from the MTA and store it in the Cyrus mail 
format, and the ctl_cyrusdb binary to maintain Cyrus’s 
extensive collection of mail metadata. The system also 
requires the primary SASL library libsasl2, as well as 
some helper libraries which implement particular 
authentication mechanisms. Cyrus is configured using 
the files cyrus.conf, which specifies the listeners and 
periodic processes spawned by master, and imapd.conf, 
which configures IMAP-specific options. 


In order to support fast and scalable service, all 
Cyrus code runs under a single UNIX user account. It 
is possible to use the UNIX passwd file for authentica- 
tion of IMAP users, but doing so requires the use of a 
special daemon, saslauthd, which runs as root and with 
which the SASL authentication mechanisms commu- 
nicate via a named socket. 


The other major Cyrus design decision which 
contributes to scalability is the custom mailbox for- 
mat. Individual e-mail messages are stored in a read- 
able format in flat files, one file per message con- 
tained in one directory per mail folder. (This directory 
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structure is similar to that of Maildir.) Mail metadata 
is stored in binary database files using a Cyrus-spe- 
cific format, and thus cannot necessarily be read by 
non-Cyrus tools. 


These two design choices bring with them the 
requirement that all access to or support of user mail 
be performed using Cyrus tools, and typically over the 
network. As a result, mail administration commands 
are provided via the remote IMAP interface, and a 
special subsystem, called sieve,’ is provided for end 
user mail filtering tasks which other systems handle 
by allowing users to install procmail filters. When 
installed, sieve adds extra code, as well as some mech- 
anism for end users to edit their filters — the most com- 
mon is an extra port listener through which users can 
authenticate and make changes over the network. 


Impressions 


I found Cyrus to be the most impressive of the 
codebases in terms of layout. The code is modular and 
well documented, and can be easily compiled using 
GNU configure. 


Since the IMAP server is designed to be run in a 
black box, any functionality not explicitly provided 
should be denied. Therefore, Project Cyrus takes 
breaches very seriously. From a security perspective, 
the design feature by which Cyrus runs all its code as 
the imapd user has both a major benefit and a serious 
drawback. The benefit is that no code (other than the 
optional saslauthd) runs as root, so it should never be 
possible to gain root access on the Cyrus box solely by 
compromising the Cyrus server. Therefore, if your pri- 
mary concern is the prevention of root compromises 
on your network, the Cyrus methodology is a very 
good one. However, the downside is that any arbitrary 
code execution vulnerability constitutes a total breach 
of the IMAP system. If your primary concern is the 
safety of the mail system itself, you might prefer a 
system in which a buffer overflow in post-authentica- 
tion code gave the authenticated user access only to 
his own mail, rather than to all mail and mail metadata 
on the system. 


Cyrus provides a large number of features inter- 
nally, some of which (such as mail administration and 
filtering) are required by the design of the system, but 
some of which seem like evidence of feature creep. 
For instance, there is a notification daemon to provide 
flexible user notification of new e-mail. Additionally, 
Cyrus, like UW, has a built-in NNTP client, and pro- 
vides anonymous user access for the purpose of using 
the IMAP server as an NNTP aggregator. 


Obviously, the in-band mail administration is 
itself risky, since it opens up more code to the IMAP 
interface. This risk is somewhat mitigated by the clean 
and modular design of the IMAP-visible code, but 
cannot be entirely removed. The Cyrus system is also 


4Sieve is an open standard mail filtering language, defined 
by RFC 3028. 
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busier than the others — the master process spawns 
periodic tasks related to cleaning and optimizing the 
database, while UW and Courier’s daemons perform 
no work unless a connection is being served. 


Courier-IMAP Server 
Overview 


Courier-IMAP is the IMAP server component of 
the Courier MTA [2]. The IMAP server is packaged 
with the MTA, but is also available and configurable 
as a stand-alone server. Courier is designed to be fast 
and scalable while being interoperable with standard 
UNIX permission schemes and mailbox formats. It 
attains this balance by using the Maildir format, which 
is a one-message-per-file format similar to Cyrus, but 
is supported by many other applications, including 
several MTAs. 


Courier-IMAP has been in development since 
1998, and has been a continuous codebase since that 
time. Like Cyrus, Courier is divided into an IMAP 
portion of the codebase and an authentication library, 
called courier-authlib. However, the imap/authlib split 
was recent in this case, dating back only as far as the 
2004 release of courier-imap-4.0.0. 


The current codebase contains 135,000 lines of 
code and 550,000 lines of other files, so it is relatively 
lean and also well documented. Approximately 30,000 
lines of code are represented by the authentication 
library, and the rest by IMAP. The IMAP codebase 
includes various compiled-in helper libraries, such as 
Unicode handlers, support for various mail-relevant 
RFCs, and support for the Maildir format. The dae- 
mon code itself is divided into: a network listener 
which provides the functionality of inetd and tcpwrap- 
pers in a Courier-specific fashion (couriertcpd), a binary 
designed for handling unauthenticated IMAP connec- 
tions (imaplogin), and a binary which should only ever 
be run by authenticated IMAP connections (imapd). 


In addition, Courier-IMAP requires its own sys- 
log frontend (courierlogger), several libraries related to 
authlib, and a mandatory authentication daemon (auth- 
daemond), which listens on a socket. (By contrast to 
Cyrus, Courier requires authdaemond even if UNIX 
passwords are not used for authentication.) 


The Courier server has as a design goal protect- 
ing the system from its own end users (legitimate or 
otherwise). Therefore, it has a number of configura- 
tion settings related to limiting the system resources 
available to IMAP users. 


Impressions 


Courier-IMAP has many positive security-rele- 
vant features, and, at an overview level, seems very 
well designed. The code is relatively minimal in many 
important ways — no NNTP client or anonymous access 
is provided. Privilege separation is cleanly implemented 
at a conceptual level. For instance, Courier requires a 
separate configuration file to run imapd on its non-SSL 
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port and on its SSL port, so it is very easy to configure 
only the secure port with no risk of accidentally 
enabling insecure access. Also, the login design, in 
which the pre-authenticated IMAP connection is han- 
dled by an entirely separate binary from the one which 
provides logged-in functions, is very clean. 


Upon closer inspection, however, many things in 
the code disappoint. To increase efficiency, the imaplo- 
gin process does not always terminate on unsuccessful 
connections. The way in which this is implemented 
potentially opens opportunities for the connection to 
be left in a partially logged-in state. A bug in this por- 
tion of the code might allow an attacker to circumvent 
the privilege separation. In addition, the code layout 
itself is hard to follow and inconsistently modular. 


Compilation and configuration were difficult and 
seemingly broken — configure is run once for each sub- 
directory of the compile directory (rather than main- 
taining a cache of, for instance, the location of gcc, or 
the size of an integer), leading to slow and repetitive 
compilation. More seriously, the compile-time config- 
uration takes some values from outside the configura- 
tion directory in an undocumented manner. Running 
make clean in the Courier directory removes files 
which are required for compilation, so it is necessary 
to restore from the original tarball in order to redo a 
compilation. None of these items is relevant to the 
running system, per se, and each may be attributable 
to the recent imap/authlib code split, but they are 
inconvenient and do not inspire confidence in the 
package design. 


In addition, there are some features which do not 
seem like good ideas. The cyrus-sas! implementation 
which allows non-daemon authentication mechanisms 
seems preferable to the courier-authlib implementation 
which always requires a daemon, thereby mandating 
that the system be open to attacks against the daemon 
itself. The server also provides some strange user fea- 
tures. Most notable among these are: INBOX.Outbox, 
a special user-visible mailbox into which users can 
place messages for immediate transmission via SMTP, 
and loginexec, a file within a user’s top-level Maildir 
which is always executed and removed, regardless of 
ownership, if it exists. 


Summary of Software Observation Results 


It is certainly the case that manual software 
observation gives an incomplete impression of soft- 
ware security. The overall code layout, modularity, 
and privilege separation are obscured from the time- 
constrained examiner, who sees only the set of files 
installed and the modularity of the main() loop. 


The most visible features are this high-level view 
of the code, the ease of installation and compilation, 
and the presence and quality of documentation (partic- 
ularly the subset used to assist the initial installation). 
It is easy to find the set of mandatory requirements 
and dependencies of each package in the chosen 
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configuration, but impossible to tell what additional 
requirements other configurations might impose. 


In addition, any extra features which are offered 
to the user or provided in the software’s back end are 
very visible. If such features appear to be security 
risks, their effect on the examiner’s opinion of the 
code may be disproportionate to their effect on the 
actual code security. In this particular case, my views 
on the codebases were strongly affected by the defen- 
sive attitude towards security concerns taken by the 
UW-IMAP documentation, and by the presence of 
strange features in the Courier-IMAP codebase. 


Based on manual software observation alone, 
Cyrus appeared likely to be the most secure and UW 
likely to be the least secure of the three packages. 


Entry/Exit Point Analysis 


In this section of the paper, I return to the formal 
definition of attack surfaces, and describe steps taken 
towards a methodical measurement of the attack sur- 
faces of my testbed IMAP servers. I report on the par- 
tial results obtained from my analysis, and discuss 
how the results compare to those derived by manual 
examination. This comparison is useful both because 
of what it says about IMAP servers per se, and 
because, in order to use entry/exit point analysis to 
measure attack surfaces, we need to ensure that the 
results obtained from that analysis are meaningful. 


In addition, my analysis is also interesting in 
terms of the obstacles to automated measurement 
which I encountered. The problems encountered in 
this analysis, some technical and some related to the 
analysis methodology, need to be solved in order to 
meaningfully automate the measurement of the attack 
surface of a given codebase. 


Entry/Exit Background 


The entry/exit methodology endeavors to provide 
two major developments in the measurement of attack 
surfaces. The first is the ability to automate the dis- 
covery of attack surface elements. The second is the 
ability to numerically compare the attack surfaces of 
different codebases. 

Automated Discovery of Attack Surface Elements 

Entry/exit analysis utilizes the following simple 
insight into attack surface measurement: In order to 
launch an attack on a system, an attacker must either 
transmit data into the system or receive data from the 
system [13]. 

Suppose we want to look at a codebase and, at 
the code level, find all the places which might be part 
of the attack surface. Any place in the code which is 
part of the attack surface must contain a call which 
receives data from or transmits data to the outside of 
the system. Therefore, if we find all such points, then 
we have found all positions on the attack surface, and 
we need only verify that each such point is actually an 
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attack surface element, and classify each one into an 
attack class (a set of attack surface elements which 
pose the same level of risk to the system). 


This opens the door for automated attack surface 
measurement, since it should be possible to find entry 
and exit points in a manner which is at least somewhat 
automated. 


Numerical Comparison of Attack Surfaces 


One major difficulty with the attack surfaces dis- 
covered by a more ad hoc analysis is that they cannot 
be compared to one another, because there is no clear 
way to numerically order the features measured. How- 
ever, if we use entry and exit points to find positions 
within the code at which attack surface elements 
occur, our task becomes easier. 


Suppose that, for each position in the code, we 
can determine what access rights are needed to run 
that code, and what privileges the running code has. 
Then, we have a one-dimensional set of access rights 
which are directly comparable, and a similar set of 
privileges. Moreover, the relative ordering we should 
use on these elements is clear: it is more difficult to 
gain administrator access than it is to gain user access, 
and it is more difficult to gain user access than it is to 
gain unauthenticated access. Similarly, root is more 
powerful than an imapd system user, which is more 
powerful than a normal user, which is more powerful 
than a service account. 


We can then assign to privilege and access levels 
numerical values which are consistent with this order- 
ing. Given these, we compute the attackability of a 
given attack surface item by observing that higher priv- 
ilege items are more desirable targets (more attackable), 
and that higher access-right items are more difficult to 
target (less attackable). If we have defined functions 

p: {privileges} > % 
and 

ac : {access rights} > % , 
and if a given item has privilege O and access rights 
B, then we can define the attackability of that item as 
BQ) g 
ac(B) 
Therefore, if we have two different items with compa- 
rable privilege and access levels, we can numerically 
compare them in terms of attackability, and we can 
sum over all the attackabilities in a given codebase’s 
attack surface, and compare the sum to that of another 
codebase. 


attackability = 


However, we cannot actually measure attackability 
as a one-dimensional quantity. This is because the attack 
surface, as noted in the introductory sections, consists of 
three different types of elements — methods, data, and 
channels — and there is no straightforward way to 


5This definition is arbitrary, and is chosen because it is a 
simple function which varies directly with privilege and in- 
versely with access rights. 
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correlate elements of these three types. Therefore, 
entry/exit analysis should ideally yield a three-dimen- 
sional vector describing the system attackability, data 
attackability, and channel attackability of a codebase. 


Methodology for Entry/Exit Point Measurement 


In this section, I discuss both the methodology I 
used for the successful measurement of method entry 
and exit points in the IMAP codebases, and my efforts 
to measure data and channel entry and exit points. 


Measuring Data and Channels 


The insight in measurement of data and channels 
is that, because of the way UNIX is designed, a process 
cannot read or write to a channel or persistent data 
object without making either a system call or a call to 
an external library which makes that system call itself. 


Of course, we do not know the identities of all 
those external system or library calls. However, they 
are easy to find: a binary examination tool such as nm 
[8] will find the list of symbols which are referenced 
by a piece of compiled code. Then, a source code 
examination tool such as ctags [6] will give us a list of 
symbols actually defined within the code. Any sym- 
bols which show up in the nm output but not in the 
Clags output must be externally defined. 


Once discovered, these external methods must be 
examined manually to determine whether they access 
data or channels, and what manner of access they per- 
form (read, write, create, delete). However, in my 
experience, there were only about 150-200 unique 
external methods per codebase. Even better, these 
methods are provided by external libraries or by the 
operating system, rather than by the examined code- 
bases. Therefore we can expect them to behave identi- 
cally across codebases. Once the behavior of fopen() 
has been classified on a given operating system for one 
codebase, that classification can be used without modi- 
fication for any other codebase running on that operat- 
ing system. I was able to classify the calls made by my 
sample codebases, and to obtain a list of internal meth- 
ods which made calls to external data and/or channels. 


From there, the task became more difficult. The 
UW-IMAP codebase contained 896 instances in which 
an internal function made an external call involving 
data or channel access. That number might correspond 
to even more actual references. (For instance, a modu- 
lar codebase might define an internal function whose 
job was to open a file and read from it. That function 
might be referenced from multiple different places in 
the code, each time with a different filename.) A very 
good code analyser might be able to find internal 
object names, leaving us with statements of the form 
“routine X read from the file whose name is stored in 
the variable foo at line 796.” However, I did not have 
access to a code analyser of that quality, and, even so, 
this tells us nothing about which entities outside of the 
given codebase require permission to read or write the 
file. nor about whether the codebase’s installation 
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procedure faithfully adheres to the strictest permis- 
sions possible rather than negligently substituting 
more relaxed permissions than needed. 


We can instead work from the other end, and use 
software observation in a jailed environment to obtain 
the full list of files that each codebase requires in 
order to operate. However, this tells us only about the 
files required by the specific configuration used in the 
example jail, rather than the full set of files which 
might be required or used by the software. 


Thus, using entry and exit points to count data and 
channel attack surface elements is not yet a solved problem. 


Measuring Methods 


This problem is more tractable. We would like to 
measure the set of all internal code which is runnable 
by an attacker. Provisionally, we measure code at the 
function level. First, we look at code which the 
attacker can call directly. Each of the three IMAP dae- 
mons uses a function with a name like main() to 
receive all input from the remote user, meaning that 
each daemon provides one reachable function. This is 
not very interesting. 


Therefore, we can extend our analysis by looking at 
functions which are themselves called by those directly 
reachable functions, and are thus indirectly reachable by 
the attacker. We obtain these function names using a pro- 
gram flow analysis tool. In the case of IMAP analysis, I 
used a tool called cflow [7]. Display 1 shows some sam- 
ple cflow output from the UW-IMAP case. 


This gave several hundred lines of output per 
codebase, the name of every function reachable from 
the main() function. In order to count each method in 
the attack surface, I needed to find out what access 
rights it had, and with what privilege it ran. Some 
manual preparation was needed in order to obtain that 
information from the codebase. 


On a UNIX system, the privilege with which the 
code runs is the privilege with which it started, unless 
a system call has occurred which changes privilege, 
such as setuid(). If a program starts running as root and 
later drops privilege, then any functions called before 
the setuid() call are called as root, and any functions 
called after setuid() are called as an unprivileged user. 


It is slightly more difficult to determine the 
access needed to reach a certain element of the code, 
because it is necessary to find each code location at 
which authentication is performed. For instance, a call 
which compared the password provided by the user to 
one retrieved from a local file would constitute a 
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change of authentication state — before that compari- 
son, a user is unauthenticated, but after the compari- 
son is successful, he is authenticated. 


However the access is determined, once we know 
where the access and privilege changes occur, we can 
edit the directly reachable routine, creating a copy for 
each access and privilege pair which contains only those 
calls reachable from that level. Here is an example from 
my edit of the Cyrus codebase. Display 2 shows the 
original code fragment, from the part of the main loop 
which handles the AUTHENTICATE command. 


In Cyrus, the variable imapd_userid is always set upon 
successful authentication, and at no other time. I edited the 
code fragment shown in Display 3 with that in mind, retain- 
ing only the code accessible to an unauthenticated user. 


Note that I needed to verify that it is possible for 
authenticate_command() to return to the main loop upon 
failure. If authenticate_command() instead killed the 
process, or jumped to another segment of code, then I 
would have considered the call to snmp_increment() to 
also be unreachable by an unauthenticated user, and 
would have deleted it as well. 


Once the edited copy had been made, I could then 
rerun Cflow and obtain a subset of the original code ele- 
ments, the subset reachable by the unauthenticated user. 
I then repeated this process for the Cyrus authenticated 
user, for the anonymous user, and for the administrative 
user, and for all combinations of access and privilege in 
each of the UW-IMAP and Courier codebases. 


Obstacles to Entry/Exit Point Measurement 


In the process of counting reachable methods, I 
encountered a number of obstacles, some technologi- 
cal, some integral to the enumeration process. I men- 
tion both types of difficulties, since technological 
problems might be solved by better tools, but, if those 
tools do not exist, the process of writing them offsets 
the time gains of automation. 


Determining Whether Data Has Been Transmitted 


In principle, we are not concerned with all func- 
tions reachable from the direct entry point, but only 
with functions through which the attacker can transmit 
the data required by his attack. How do we measure 
the subset of functions which actually allow the 
attacker to transmit data? 


Clearly, if an attacker can input arbitrary data 
which is immediately read into an internal buffer 
belonging to the function, he has transmitted data into 





main {src/imapd/imapd.c 253} 
strrchr {} 


fatal {} 


mail_parameters ... {3} 
free {} 


ONDUNFWNHeE 


mail_parameters {src/c-client/mail.c 297} 


env_parameters (src/osdep/unix/env_unix.c 150} 
fs_give {src/osdep/unix/fs_unix.c 57} 


Display 1: Partial cflow output from UW-IMAP codebase. 
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the function. However, consider a more restrictive case, 
in which, for instance, the input must have a certain for- 
mat, such as being a valid SSL key, but is otherwise 
chosen by the attacker. Then consider the case in which 
the attacker can only insert a single integer value, such 
as a return code. Consider the case in which the 
attacker’s return code is ignored by the calling function. 


My conclusion was that the safest route is to 
count every called function, regardless of the syntax or 
content of the call. Display 4 shows an example which 
supports that conclusion. It is taken from the UW 
imap-2004a codebase’ which was reported to be vul- 
nerable in early 2005 [10] (all code irrelevant to the 
attack has been excised for clarity). 


The problem here is caused by the logic used to 
set the variable u — the check done on md5try is 


SCyrus-IMAP is copyright 1994-2000 Carnegie Mellon 
University. Some function and variable names have been 
modified at the request of CMU. 

TUW-IMAP is copyright 1988-2004 University of Wash- 
ington. 


if (imapd_userid) { 
protocol_printf(imapd_out, 
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backwards, so that, if md5try is 0 (if the attacker has 
tried to authenticate too many times using CRAM- 
MDS), the call automatically succeeds where it should 
automatically fail. The attacker does not need to send 
any particular exploit data in order to break into this 
system. All he needs to do is to attempt CRAM-MDS5 
authentication unsuccessfully three times, and then he 
will be authenticated on the fourth try. 


In that light, it seems very reasonable to claim that 
causing a line of code to be run is sufficient, from an 
entry/exit point perspective, to transmit data into that code. 


Undercounts and Overcounts 


In order to automatically enumerate reachable 
methods, it is necessary to have an accurate method of 
detecting code execution paths. There are many tools 
which profess to do this, but, in practice, there are 
problems which lead to undercounts or overcounts. 


First, in order to avoid undercounts, it is neces- 
sary to have a program which parses the code accu- 
rately. I used cflow for my final analysis because it 
employs gcc -E, and therefore works quite well. 


"%s BAD Already authenticated==>[ignored: r]<====>[ignored: n]<==", tag.s); 


continue; 


} 


authenticate_command(tag.s, argl.s, haveinitresp ? arg2.s 
snmp_increment (AUTHENTICATE_COUNT, 1); 


} else if (!imapd_userid) goto nologin; 
else if (!strcemp(command.s, "Append") ) 
if (c !=’° ’) goto argsmissing; 


NULL) ; 


Display 2: Original Cyrus-[MAP authentication code fragment.® 


if (imapd_userid) { 


} 


authenticate_command(tag.s, argl.s, haveinitresp ? arg2.s 
snmp_increment (AUTHENTICATE_COUNT, 1); 


} else if (!imapd_userid) goto nologin; 


NULL) ; 


Display 3: Modified Cyrus-I[MAP code: subset of code accessible to an unauthenticated user. 


static int md5try = 3; 


char *auth_md5_server (authresponse_t responder,int argc, char *argv[]) 


{ 
char *ret = NIL; 


u = (md5try && strcmp (hash,hmac_md5 (chal,cl,p,pl))) ? NIL : user; 


if (u && authserver_login (u,authuser,argc,argv) ) 


ret = myusername (); 
else if (md5try) --md5try; 


if (!ret) sleep (3); 
return ret; 


/* slow down possible cracker */ 


Display 4: Vulnerable CRAM-MDS function in UW imap-2004a. 


18 19th Large Installation System Administration Conference (LISA ’05) 


Golubitsky 


However, cflow ignores function arguments and pro- 
gram logic, which one might prefer to take into 
account. 


For instance, in the following example, method_c 
is not actually reachable from method_a, but cflow 
would claim it was: 

method_a() { 
method_b (DONT_USE_X) ;: 
} 


method_b(int flag) { 
if (flag != DONT_USE_X) { 
method_c(); 
} 
} 


Other tools which perform some of these func- 
tions include doxygen [5], which parses function argu- 
ments in a relatively usable fashion, but has a poor 
understanding of C syntax and does not deal with 
logic, and cqual [3], which is intended to handle both 
function arguments and program logic, but is difficult 
to work with and possibly buggy in practice. 


A major source of undercounts in the IMAP case 
is caused by function pointers — variables which store 
the names of functions. At the point of execution, the 
function is referenced only by the name of the vari- 
able, so there is no reliable way to tell what actual 
function is being executed. All three IMAP codebases 
make use of function pointers, and no tool I found was 
capable of parsing them. 


Classifying Multiply-Accessed Methods 


Once the codebase has been divided by privilege 
and access levels, many functions are identified which 
are accessible by code running at different levels. This 
code may belong to internal or external helper rou- 
tines, or it may be authentication-relevant code which 
has inadvertently been left too open. In order to clas- 
sify the code, I needed to generate a strategy for han- 
dling such functions. 


One strategy would be to count each function 
twice, once per privilege/access pair from which it 
was accessible. However, this seemed to overcount in 
the case of, for instance, an administrative user and a 
normal user who could both access normal user code. 


Another approach would be to count only the 
worse of the two levels, i.e., if the function was acces- 
sible to either an unauthenticated user or to an authen- 
ticated user, count it as unauthenticated. However, this 
would probably provide an undercount, because an 
authenticated user might be able to reach more of the 
functionality of a routine than an unauthenticated user. 
If there are two attack paths related to a given func- 
tion, both should be counted. 


I compromised by creating new privilege and 
access levels to reflect multiply-accessible code. For 
instance, if root and user are privilege levels, then 
roottuser would be the privilege level for code which 
could be reached at either of those levels. In the IMAP 
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codebases, the appropriate total orderings of privilege and 
access levels were clear even with the additional levels. 


Results of Entry/Exit Point Measurement 


For each codebase, I identified the function 
responsible for receiving network connections from 
remote users, and broke down the codebase by privi- 
lege and access levels as previously described. For the 
UW and Cyrus codebases, this was a simple matter of 
finding the single input-handling function. 


The Courier codebase was somewhat more com- 
plex. In that case, the program courertcpd receives and 
processes an incoming network connection, does some 
checking, and then hands the connection off to imaplogin, 
which processes authentication routines. If imaplogin 
reports success, then an imapd process is spawned. 
Therefore, in order to find all the routines which could 
run at each privilege and each access level in the Courier 
case, I needed to analyse all three of these programs. 


I also performed an analysis of Courier in which I 
omitted the couriertcpd code, which all runs as root and 
is accessible to unauthenticated users. Since the Courier 
and Cyrus codebases include network listeners in their 
codebases, while UW relies on inetd to provide network 
connectivity, I was interested in gauging the amount of 
code required to listen for network connections. 





Fullcode Runsas_ Accessible 
root by any 


Figure 2: Number of reachable methods in each 
codebase: (a) full codebase, (b) code which runs 
as root, (c) code accessible at any access level. 


Method Counts By Codebase 


Figure 2 shows a representative subset of break- 
downs of method counts among the codebases. The 
left graph shows the total number of methods reach- 
able in each codebase. The middle graph shows the 
number of methods reachable by code which is run- 
ning as root. (Note that Cyrus has no reachable code 
which runs as root.) The right graph shows the number 
of methods which are open to all access levels defined 
for the codebase. That is, in the UW case, this graphs 
functions which are reachable by an unauthenticated 
user, by an anonymous user, and by an authenticated 
user. In the Cyrus case, this graphs functions which 
the unauthenticated, anonymous, authenticated, and 
administrative users can all reach. In the Courier case, 
this graphs functions which the unauthenticated and 
authenticated users can both reach. 
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Total Orderings for Privilege and Access Rights 

Once I had counted all reachable code methods, I 
needed to populate the total orderings for the privi- 
leges and access rights. All types in each ordering 
needed to be assigned numerical values, so that the 
attackability of each attack class, and thus of the entire 
system attack surface, could be computed. 


Access Rights | Points | 


fadmin | 
fanon 








1.5 


Funauth SSS 
auth Funauth ~~~ 
‘auth anon +unauth | 08 


Table 1: Attackability ordering for method access 
rights (higher value is harder to attack). 


/Privitege | Points 
sevice SYS 
fuser ——SSidYSC—* 










[root Fuser Pservice | 14 


Table 2: Attackability ordering for method privileges 
(higher value is more valuable target). 





Table | contains the ordering I derived for access 
rights, and Table 2 contains the ordering for privi- 
leges. Note that the ordering results naturally from 
known information about security of UNIX servers 
and the IMAP domain, but that the specific numerical 
values chosen are arbitrary. 


System Attackability of IMAP Servers 


Table 3 shows the system attackability of each 
IMAP server given these values and the reachable 
methods determined by the automated method. This 
table shows that, according to the attackability metric 
used here, Courier is the least vulnerable of the 
servers, while UW and Cyrus score similarly. The 
results also indicate that much of Courier’s vulnerabil- 
ity is caused by its network listening code. This 
implies that Cyrus, which also contains network lis- 
tening code, might be less vulnerable than UW were 
the functionality provided to be taken into account. 


However, Figure 3 is worth noting. It demonstrates 
that the attackability metric is somewhat dependent on the 
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exact numbers of points chosen for various privileges and 
access rights, even if the total ordering is held constant. 
By default, I assigned seven points to the special imapd 
user employed by Cyrus. However, a site which was 
most conscious of the need to avoid root compromise on 
their machines could assign imapd five points, close to 
the value for an unprivileged user, and would find that 
Cyrus significantly outperformed UW in attackability. A 
site with a different posture could note that any attack on 
the imapd user risked opening up all users’ e-mail, and 
therefore was almost as bad as a root compromise. If this 
site assigned nine points to imapd, Cyrus would suddenly 
appear radically more attackable than UW. 


codebase _ |r 
Codebase Attackability 


Table 3: System attackability of IMAP servers. 
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imapd=5 imapd=9 noanon 
points points access 


Figure 3: System Attackability of IMAP servers, with 
(a) imapd user worth 5 points, (b) imapd user 
worth 9 points, (c) imapd user worth 7 points and 
no anonymous access. 


This tradeoff between protecting root and protect- 
ing user mail from other users is symptomatic of an 
obvious security consideration which arises when trying 
to decide whether to use Cyrus or a more convention- 
ally-designed IMAP server. This example is included to 
demonstrate that the attackability metric does not protect 
us from having to make that judgment call. 


In general, however, the attackability metric seems 
to agree reasonably well with observation. Despite the 
large size of the Cyrus codebase, its attackability is sim- 
ilar to that of UW-IMAP, indicating that Cyrus has good 
privilege separation while UW-IMAP does not. Indeed, 
the major effect of this metric as applied is to reward 
code with good privilege separation, which is a reason- 
able goal, though only one of many. 


The couriertcpd example demonstrates that it is 
difficult to compare systems which perform different 
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functions, and that even between systems which are 
very similar in functionality, one system may provide 
many more of its own dependencies. In the process of 
measuring attackability, it is necessary to consider the 
external systems on which each codebase depends. 


Conclusions 


In this section, I discuss the feasibility of per- 
forming an automated entry/exit analysis as a means 
of comparing software security. I also discuss future 
work needed to determine whether entry/exit analysis 
of attack surfaces will be practical, and whether it will 
be useful. I close with a few words about the security 
of the specific IMAP servers I studied. 


Feasibility of Entry/Exit Analysis 


One important point is that it is difficult to ensure 
that any analysis is comprehensive. In the particular case 
of this analysis, I looked at only the code available via the 
network API, and did not examine the code added by 
periodic cleanup processes, LMTP, etc. I also did not look 
at the authentication daemons provided by Cyrus and 
Courier as part of the authentication facility. IMAP com- 
municates with those via socket, so they would have sur- 
faced had I done a channel analysis. It is still necessary to 
manually examine the code’s functionality in some detail 
in order to find starting points for automated analysis. 


In addition, entry/exit analysis is not trivial to 
perform. Approximately four to eight hours of labor 
was required to prepare each codebase for analysis by 
cflow, and that took into account significant familiar- 
ity with the code in question. The most time-consum- 
ing aspect is dividing the code based on privilege and 
access levels, but some time is also required to find 
the correct internal function definitions within a com- 
plex codebase, given that cflow and related tools can- 
not read Makefiles. 


The division of the code is not entirely automatable 
work — some judgment calls are required in order to 
determine, for instance, at exactly what point authentica- 
tion takes place. In theory, the Courier approach, in which 
a different physical piece of software handles execution at 
each privilege and access level, should make things eas- 
ier. In practice, there are thorny issues even in that imple- 
mentation, since authentication, setuid(), and exec() of the 
other daemon do not all take place simultaneously, nor 
even necessarily within the same function as one another. 


In addition to all this, | did not attempt the data or 
channel enumerations, which would be more complex 
and difficult. In those cases, it might be complicated to 
even determine the total ordering among access rights — 
is a channel which can restrict by IP addresses higher or 
lower risk than a channel which implements anti-DoS 
protection? 

However, entry/exit analysis holds the promise 
of giving usable information which speaks meaning- 
fully about the internal security posture of a codebase, 
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and which is derived from less work than that required 
for a comprehensive manual analysis. 


Future Work 


Measurement of system attack surfaces could be 
improved by viewing the code at a level other than the 
function level, i.e., by looking inside functions and 
giving more weight to larger or more complex ones in 
analysis. This would be aided by the use of better tools 
which could trace the flow of execution through the 
code more accurately. It would also be useful to take 
into account which functions were actually compiled 
into the code in use, perhaps by comparing the full 
reachable code tree with symbol analysis of the result- 
ing binary, or by replacing cflow with some tool which 
actually could read Makefiles. 


Performing a data and channel analysis of entry 
and exit points would also provide a significant amount 
of information about the practicality of using entry/exit 
analysis to obtain a full picture of the system. 


In addition, the problem of comparing codebases 
which perform the same overall function, but not the 
same subsets of that function, will need to be solved. 
For instance, if it were possible to determine what sub- 
set of the Cyrus and Courier codebases was dedicated 
to providing port listener functionality, then administra- 
tors could compare an analysis of inetd to just those 
subsets, and thereby determine whether the Cyrus and 
Courier implementations of that functionality were 
more or less risky than a third-party implementation. 


Comparison of IMAP Server Software 


I will finish with some brief words about the rel- 
ative security of IMAP servers as a result of this anal- 
ysis. My subjective impression that the Cyrus and 
Courier codebases are better designed than the UW 
codebase was corroborated by the entry/exit analysis 
results. In particular, the lack of internal privilege sep- 
aration in UW-IMAP indicated that that codebase is 
not designed in a security-conscious way. In compar- 
ing Cyrus and Courier, I determine that Cyrus is better 
built, but overreaches significantly in terms of the 
number and diversity of features it offers, while 
Courier is designed in a very security-conscious man- 
ner, but implemented somewhat more strangely. My 
conclusion is that despite the administrative headaches 
it introduces, Courier is likely to be the best security 
risk when choosing one of these three products to act 
as an open source IMAP server. 
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Because much of my entry/exit analysis was per- 
formed manually, there is no distributable software 
associated with this project. The project page at http:// 
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scripts I used, and the detailed output of the analysis. 
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ABSTRACT 


User-mode resource hiding through API interception and filtering is a well-known technique 
used by malware programs to achieve stealth. Although it is not as powerful as kernel-mode 
techniques, it is more portable and reliable and, as a result, widely used. In this paper, we describe 
the design and implementation of a fast scanner that uses a cross-view diff approach to detect all 
user-mode hiding Trojans and rootkits. We also present detection results from a large-scale 
enterprise deployment to demonstrate the effectiveness of the tool. 


Introduction 


The term “rootkit” generally refers to the class 
of stealth malware programs that hide “resources” 
from the operating system resource-enumeration APIs. 
For example, a rootkit may be used to hide critical 
executable files from anti-virus scanners, to hide criti- 
cal Windows Registry entries from experienced sys- 
tem administrators using programs such as RegEdit, 
and to hide critical processes from average users run- 
ning Windows Task Manager. By making critical 
resources “invisible” to the APIs and the system utili- 
ties that make use of such APIs, rootkits have a much 
better chance of evading detection and maintaining 
full control of infected machines for an extended 
period of time. 


The techniques used by rootkits to achieve 
stealth can be broadly divided into two categories. 
Rootkits in the first category intercept resource-enu- 
meration APIs in user mode or kernel mode, and 
remove selected entries before the results are returned 
to the API caller. Rootkits in the second category per- 
form Direct Kernel Object Manipulation (DKOM) to 
remove selected resource entries from a cached list 
(such as the ActiveProcessList on Windows) that is 
designed specifically for answering resource queries 
and not critical to the actual functions of the resources 
or to the functioning of the operating system. 


Cross-View Diff-Based Rootkit Detection 


The traditional signature-based anti-virus 
approach cannot effectively deal with rootkit infec- 
tions in the enterprise for three reasons. First, viruses 
usually have well-defined bundles and scope of 
impact, which can be analyzed in a lab to generate 
fixed signatures. In contrast, rootkits are merely 
“resource hiders” that can be used to hide any hacker 
tools, keyloggers, spyware programs, FTP servers, 
etc., so each rootkit infection can potentially involve a 
customized bundle with a different scope of impact. 


Second, sophisticated hackers who attack large 
enterprise are less likely to use common tools or 


malware programs for which commercial anti-virus 
scanners already have signatures. Finally, while the 
major strength of anti-virus software is to detect and 
automatically remove known-bad malware programs 
without user intervention, corporate security organiza- 
tions in large enterprises often need to investigate 
every rootkit infection case to assess potential dam- 
ages and prevent future infections; automatic removal 
is often not desirable. 


We previously proposed a non-signature, diff- 
based approach to rootkit detection, called Ghost- 
Buster [WVR+04]. The basic idea is to get “the lie” 
from inside the box, get ‘the truth” from outside the 
box, and take a diff to detect hidden resources. Specif- 
ically, we get “the lie’ by enumerating files and Reg- 
istry entries through infected APIs inside the operating 
system. Then we boot into a clean CD and scan the 
files and Registry on the infected drive as a data drive. 
Since the rootkit is not running, we obtain “the truth” 
that includes the resources that the rootkit was trying 
to hide so that the diff between “the lie” and “the 
truth” will reveal precisely those hidden entries. Such 
a diff-based approach essentially turns the hiding 
behavior into its own detection mechanism and turns 
one of the most difficult anti-malware problems into 
one of the easiest problems to solve. 


Fast User-Mode Rootkit Scanner for the Enterprise 


Although this CD-boot-based solution can cover 
a broad range of rootkits, no matter how they are oper- 
ating in user mode or kernel mode, it is inconvenient, 
requires user cooperation, and is difficult to deploy on 
an enterprise scale as a scanner. Since the statistics 
from a major Product Support Service (PSS) organiza- 
tion indicates that user-mode rootkits account for over 
90% of the reported enterprise rootkit cases, it is desir- 
able to have a scalable rootkit scanner that can be 
deployed in the enterprise to detect all user-mode 
rootkits, which intercept and filter resource API calls 
in the address space of each user-mode process [YH03, 
YNO04). 
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We have developed such a rootkit scanner for 
Windows platforms. It is based on the key observation 
that, when considering only user-mode rootkits, ‘the 
truth” can be obtained from the lowest level of user 
mode by properly preparing the call stack parameters 
and using a few lines of assembly code to directly 
invoke the transition into the kernel, without going 
through the regular user-mode Win32 API code. In our 
current implementation, we use the difference to detect 
hidden processes and to detect hidden hooks to Auto- 
Start Extensibility Points (ASEPs), which are those 
Registry locations most frequently attacked by spyware, 
Trojans, and rootkits based on an extensive study 
[WRV+04, WBV+05]. If any hidden processes or ASEP 
hooks are detected, we then look for potentially hidden 
files associated with those hidden entries. This allows us 
to detect user-mode rootkit infections in a few seconds, 
without requiring a reboot or any user participation. 


Implementation 


Figure | (a) and (b) illustrates how several real- 
world Trojans and rootkits hook into the Registry and 
process enumeration API calling chains, respectively, 
to hide their resources. Urbin and Mersting are Trojan 
DLLs that make modifications at the highest level by 
altering the per-process Import Address Table (IAT) 
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entries of the Registry enumeration APIs to point to 
their Trojan functions (an IAT contains pointers to 
functions exported by loaded DLLs [HB05]). In con- 
trast, Vanquish directly modifies the loaded, in-mem- 
ory API code to interject its code. Both techniques 
cause the Trojan functions to appear in the call stack 
trace of a kernel or user-mode debugging session. 


To achieve better stealth, Aphex and Hacker 
Defender modify the in-memory API code with a 
jump to the Trojan code along with a Trojan code 
jump back to the next instruction after the API detour 
[HB99]; the Trojan code modifies the return address 
on the stack to cause its code to be executed in the 
return path. The only difference is that Aphex modi- 
fies the RegEnumValue API code inside Apvapi32.dll 
(denoted by Advapi32!RegEnum Value), while Hacker 
Defender modifies the lower-level NtEnumerateKey 
API exported by NtDIl.dll. YYT rootkit operates very 
similarly to Hacker Defender. ProBot SE in Figure 
l(a) and FU in Figure 1(b) are kernel-mode stealth 
programs that cannot be detected by the scanner 
described in this paper. 


Our tool performs the following steps to obtain 
“the truth” from underneath all the user-mode Trojans 
and rootkits shown in Figure 1. 


RegEdit 







Urbin & 
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Can} 


User mode 
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TaskMgr 


NtDII!NtQuerySystemInformation 
Import Address Table (LAT) 


User mode 


Kernel mode 


NtDII!NtQuerySystemInformationg 
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Figure 1: Trojans and rootkits that (a) hide Registry entries and (b) hide processes. 
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1. Set up the user-mode stack with the parameters 
required by the operating system; this is easily 
achieved by creating a function with a signature 
that exactly matches the desired Native API, 
such as NrDIl!NtEnumerateKey() and NtDIl!Nt 
QuerySystemInformation(); 

2. Populate the EAX register with the index that 
indicates to the operating system which system 
function you wish to call; 

3. Populate the EDX register with a pointer to the 
user mode’s stack parameters; 

4. Execute an “int 2e” instruction to signal a ker- 
nel-mode transition; 

5. Return to the caller. 


The previous steps basically describe what the 
code in NtDil.dll does when calling into the operating 
system. Given that the steps require direct manipula- 
tion of X86 registers, a portion of the code is written 
in assembly — this is easily obtained by disassembling 
NtDil.dll and searching for the desired function by 
name. Below is an example of the N¢tQuervSystemIn- 
formation call for retrieving “the truth” of the list of 
processes: 

__declspec (naked) 

NTSTATUS 

NTAPI 

MyNtQuerySystemInformation ( 
SYSTEM_INFORMATION_CLASS 


SystemInformationClass, 














Rootkits & Trojans 









Hacker Defender 1.0 






hxdef100.exe 







hxdefdrv.sys 
















Rootkits & Trojans 
YYT Rootkit 


Hacker Defender 1.0 





Aphex 


Hidden ASEP Hooks Detected 





Urbin HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Windows\ 
AppInit_DLLs > msvsres.dll 

Mersting HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Windows\ 
AppInit_DLLs — kbddfl.dll 


YYT Rootkit HKLM\SYSTEM\CurrentControlSet\Services\NPF — npf.sys 
HKLM\SYSTEM\CurrentControlSet\Services\TSSERVER — comine.exe 
HKLM\SYSTEM\CurrentControlSet\Services\Udfs 

HKLM\SYSTEM\CurrentControlSet\Services\HackerDefender100 > 


HKLM\SYSTEM\CurrentControlSet\Services\HackerDefenderDrv 100 > 


Aphex HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run > 
<user defined name>.exe 


HKLM\SYSTEM\CurrentControlSet\Services\Vanquish — vanquish.exe 


Hidden Processes Detected 
Berbew <random name>.exe 
comine.exe 


hxdef100.exe, and any other processes with names matching the patterns 
specified in hxdef100.ini 





By default, any process with a “~~”’-prefixed name (which is configurable) 
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PVOID SystemInformation, 
ULONG SystemInformationLength, 
PULONG ReturnLength) 


asm 


{ 


mov eax, OxAD 
lea edx, [espt+0x4] 


ret 10h 


Several important points need to be made regard- 
ing the code above. First, notice the __declspec 
(naked) compiler directive. This prevents the com- 
piler from generating prolog code for the function in 
order to ensure that the user-mode stack is in the right 
form at the time of a kernel-mode transition. Second, 
the function is labeled as NTAPI to ensure that the 
right C calling convention is used (in this case 
___stdcall). Third, the value that is moved into the 
EAX register is the index into a kernel-mode function 
dispatch table that tells the operating system which 
function to call: this value is unique to the function 
and varies from version to version of the operating 
system. Fourth, the “int 2eh” instruction sends a sig- 
nal to the operating system to tell it to initiate a kernel- 
mode transition. Finally, the parameters exactly match 
the original query API because the operating system 
will pass them directly to the kernel-mode API. 

















Figure 2: Hidden resources detected: (a) hidden Registry ASEP hooks; (b) hidden processes. 
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Once the tool obtains “‘the truth” and then uses a 
regular Win32 API call to obtain “the lie”, it compares 
the two scans and declares as hidden resources those 
that appear only in “the truth”. It is possible to see 
false positives due to the creation or deletion (depend- 
ing on the order the scans are performed) of resources 
in the time window between the scans. In practice this 
is usually not an issue. Also, this is easily mitigated by 
performing successive scans and taking the intersection 
of the missing resource sets to create the final result set. 


Experimental Results 


Results from Lab Tests 


We have tested the tool in our lab against the 
seven user-mode rootkits and Trojans shown in Figure 
1. Although the implementations of these malware 
programs are quite different, our tool was able to effi- 
ciently and effectively detect all of them in a uniform 
way. Figure 2(a) and (b) show the detected hidden 
ASEP hooks and hidden processes, respectively, for 
each malware. The Urbin, Mersting, Berbew, and YYT 
Rootkit samples were captured from the wild, while 
the Hacker Defender, Aphex, and Vanquish samples 
were downloaded from the Web. The ‘“Applnit_ 
DLLs” ASEP allows auto-loading of one or more 
DLLs into every Windows-based application that is 
running in the current log-on session [AID]; the ‘“Ser- 
vices” ASEP allows installations of always-running 
services and drivers; the “Run” ASEP allows addi- 
tional processes to be auto-started near log-in time. 


Results from Actual Deployment 


We have deployed the tool on over 200,000 desk- 
top and server machines. The executable file is copied 
from a central machine to each target machine at scan 
time; the scan results are reported back to the central 
machine, and the executable file is removed. Figure 3 


Wang and Beck 


gives some examples of detected infections. They can 
be broadly classified into two categories: hiding Tro- 
jans and full-fledged rootkits. Figures 3 (a), (b), and 
(c) show three types of hiding Trojans: they were most 
likely installed by malicious Web servers that exploit 
visiting browsers’ vulnerabilities [WBJ+05]. The Tro- 
Jans in Figure 3 (a) hide their hooks to the “AppInit_ 
DLLs” ASEP, while the Trojans in Figure 3 (b) and 
(c) hide their randomly-named processes. 


Figure 3 (d), (e), (f) show three cases of infections 
with full-fledged rootkits. We make the following obser- 
vations: first, it is common for rootkits to create mal- 
ware programs that have the same or similar filenames 
as some system programs but reside in a different direc- 
tory; for example, one of the Isass.exe processes in Fig- 
ure 3 (f) was instantiated from the Isass.exe malware 
program located in the “drivers” directory. Second, 
full-fledged rootkits tend to hook the “‘Services’”” ASEP 
to install services and drivers and they tend to hide mul- 
tiple ASEP hooks and processes. 


Rootkit Investigation Tool 


Once a rootkit-infected machine is identified in an 
enterprise, it is important to investigate the hacker’s 
intention and the damage that has been done, without 
disturbing the malware because some are designed to 
erase all traces of themselves once they realize that they 
have been detected. It is therefore highly desirable to 
have a tool that allows such non-intrusive investigations. 


We have observed that configurable rootkits, 
such as the most popular Windows rootkit “Hacker 
Defender,” typically support the notion of “root pro- 
cesses”. A root process is not infected by the rootkit and 
can see “the truth’’. It is provided for the convenience of 
the rootkit users. For example, it can be very awkward 
for hackers if the resources are hidden from their tools 


Hidden ASEP Hook: HKEY_LOCAL MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Windows\ 
Value: “AppInit_DLLs” Data: ““C:\WINDOWS\system32\log.dll” 

Hidden ASEP Hook: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Windows\ 
Value: “AppInit_DLLs” Data: ““C:\WINDOWS\System32\winpgfd.dll”’ 

Hidden ASEP Hook: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Windows\ 
Value: “AppInit_DLLs” Data: ““C:\WINDOWS\System32\winpgfd.dll” 


Hidden Procs: \Device\Harddisk Volume 1\WINDOWS\system32\Hbkqjd32.exe 
Hidden Procs: \Device\Harddisk Volume2\WINDOWS\system32\Mdnpdf32.exe 
Hidden Procs: \Device\Harddisk Volume 1\WINDOWS\System32\Ppnlan32.exe 
Hidden Procs: \Device\Harddisk Volume 1\WINDOWS\system32\Ljoocp32.exe 
Hidden Procs: \Device\Harddisk Volume 1\WINDOWS\system32\Gnaeplam.exe 


Hidden Procs: \Device\Harddisk Volume1\WINDOWS\System32\elitesig32.exe 
Hidden Proes: \Device\Harddisk Volume 1\WINDOWS\system32\elitegct32.exe 
Hidden Procs: \Device\Harddisk Volume2\WINDOWS\system32\eliteyzx32.exe 
Hidden Procs: \Device\Harddisk Volume 1\WINNT\system32\eliteohl32.exe 
Hidden Procs: \Device\Harddisk Volume2\WINN T\system32\elitemfu32.exe 
Hidden Procs: \Device\Harddisk Volume2\WINNT\system32\eliteaee32.exe 


Figures 3a-3c: Actual infection cases. 
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as well; when a rootkit is used to hide a spyware 
browser add-on executable file, it gets really tricky if 
the file is also hidden from the browser process that is 
supposed to load it. With “root process” support, the 
hacker tools and the browser can be declared as root 
processes to ensure smooth operation, while the 
resources are still hidden from all the other processes, 
system utilities, and anti-malware scanners. In the case 
of Hacker Defender, ‘“‘root processes” are listed in a 
configuration file via filenames specified with regular 
expressions. 


We have developed a technique to take advan- 
tage of the support for root processes to allow non- 


Fast User-Mode Rootkit Scanner for the Enterprise 


First, the fast user-mode rootkit scanner is used to 
identify hidden processes. Very often, one or more of 
the hidden processes are also root processes (because 
these are the most critical processes for the infection). 
Suppose process “foo.exe”’ is a detected hidden, root 
process. In the second step, we make a copy of the 
command-window program “‘cmd.exe”’, rename it to 
“foo.exe”, and launch it through “start foo.exe’’. 
Now we have a command window that is a root 
process that can see all previously hidden resources. In 
particular, the “dir’? command from this window can 
see all hidden files, including the Hacker Defender 
configuration file which reveals critical information 
about the hacker’s intention. 


intrusive investigations. It works as follows [WBOS5]. 


Hidden processes detected: 2 
PID Name 


2168 \Device\Harddisk Volume 1\WINDOWS\system32\scrss.exe 
2460 \Device\Harddisk Volume 1\WINDOWS\system32\taskmgnr.exe 


Hidden service keys detected: 2 


Key: HKEY_LOCAL_MACHINESYSTEM\CurrentControlSet\Services\scrss 
ImagePath: C:\WINDOWS\system32\scrss.exe 
Key: HKEY_LOCAL_MACHINESYSTEM\CurrentControlSet\Services\tskmgr 
ImagePath: C:\WINDOWS\system32\taskmgnr.exe 
(d) 
Hidden processes detected: 11 
PID Name 
436 \Device\Harddisk Volume2\WINDOWS\system32\smss.exe 
564 \Device\Harddisk Volume2\WINDOWS\system32\Isass.exe 
1396 \Device\Harddisk Volume2\WINDOWS\system32\svchosts.exe 
1428 \Device\Harddisk Volume2\WINDOWS\system32\csschk.exe 
1744 \Device\Harddisk Volume2\WINDOWS\system32\DNTUS26.EXE 
1976 \Device\Harddisk Volume2\WINDOWS\system32\ShellExt\_\tmp\IPSSvc.exe 
248 \Device\Harddisk Volume2\ SYSTEM 1\_system\lsass.exe 
308 \Device\Harddisk Volume2\WINDOWS\smss.exe 
364 \Device\Harddisk Volume2\ SYSTEM 1\_system\system\ioFTPD.exe 
2052 \Device\Harddisk Volume2\ SYSTEM™1\_system\Isass.exe 
2164 \Device\HarddiskVolume2\ SYSTEM" 1\_system\bot\eggdrop.exe 
Hidden service keys detected: 2 
Key: HKEY_LOCAL_MACHINESYSTEM\CurrentControlSet\Services\IPSdrv 
ImagePath: \??\c:\windows\system32\shellext\_\tmp\IPSdrv.sys 
Key: HKEY_LOCAL_MACHINESYSTEM\CurrentControlSet\Services\NetSecc 
ImagePath: c:\windows\system32\shellext\_\tmp\ipssve.exe naslib.dll 
(e) 
Hidden processes detected: 4 
PID Name 
556 \Device\Harddisk Volume2\WINDOWS\system32\services.exe 
568 \Device\Harddisk Volume2\WINDOWS\system32\Isass.exe 
1768 \Device\Harddisk Volume2\WINDOWS\java\lspool.exe 
484 \Device\Harddisk Volume2\WINDOWS\system32\drivers\Isass.exe 
Hidden service keys detected: 2 
Key: HKEY_LOCAL_MACHINESYSTEM\CurrentControlSet\Services\Lspool 
ImagePath: c:\windows\java\lspool.exe 
Key: HKEY_LOCAL_MACHINESYSTEM\CurrentControlSet\Services\r_server 
ImagePath: c:\windows\system32\drivers\Isass.exe /service 


Figures 3d-3f: Actual infection cases. 
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In the third step, we launch Task Manager, 
RegEdit, anti-virus scanner, anti-spyware scanner, etc. 
from this command window. Since this root process is 
not infected it cannot infect its child processes; as a 
result, all these utilities are now running as root processes 
that can see all previously hidden resources. After the 
investigation, malware processes, Registry entries, and 
files can all be terminated/deleted through these utilities. 


Related Work 


There are two different approaches to rootkit 
detection. The first approach targets the hiding mecha- 
nism by, for example, detecting the presence of API 
interceptions [YI, ZVI, YK, YKS, YV04]. It has at least 
two disadvantages: first, it cannot catch rootkit pro- 
grams that do not use the targeted mechanism; second, 
it may catch as false positives legitimate uses of API 
interceptions for in-memory software patching, fault- 
tolerance wrappers, security wrappers, etc. The second 
approach targets the hiding behavior by detecting any 
discrepancies between “the truth” and “the lie”. For 
example, comparing the output of “Is” and “echo *” 
can detect an infected “Is” program [B99]. Our user- 
mode rootkit detector belongs to the second category. 


There is a subtle but important difference between 
the “cross-view diff’ used in our proposal and the 
more common “cross-time diff” used in Tripwire 
[KS94] and the Strider Troubleshooter [WVS03, 
WVD+03]. The goal of a cross-time diff is to capture 
changes made to persistent state by essentially compar- 
ing snapshots from two different points in time (one 
before the changes and one after). In contrast, the goal 
of a cross-view diff is to detect hiding behavior by 
comparing two snapshots of the same state at exactly 
the same point in time, but from two different points of 
view (one through the infected path and one not). 
Cross-time diff is a more general approach for captur- 
ing a broader range of malware programs, hiding or 
not; the downside is that it typically includes a signifi- 
cant number of false positives stemming from legiti- 
mate changes and thus requires additional noise filter- 
ing, which has a negative impact on usability. In con- 
trast, cross-view diff targets only hiding malware and 
usually has zero or very few false positives because 
legitimate programs rarely hide. 


Ideally, “the truth” should be obtained from 
“outside the box” to eliminate the possibility of any 
malware intervention. The WinPE-based GhostBuster 
tool [WBV+05] took such an approach to obtain “the 
truth” of the file system and Registry. The PCI-add-in 
card described in the Copilot paper [PFM+04] or the 
Myrinet NIC described in the Bookdoors paper 
[BNG+04] can be used to obtain “the truth” of the 
process list through Direct Memory Access (DMA) 
without the knowledge or intervention of the poten- 
tially infected OS. Instead of targeting comprehensive- 
ness, our user-mode rootkit detector targets efficiency, 
scalability, and ease of use with good coverage. 


Wang and Beck 


In response to the increasing popularity of stealth 
techniques among Windows malware, several rootkit 
detection tools have been released in recent months, 
including RootkitRevealer from Sysinternals, Black- 
light Rootkit Eliminator from F-Secure, and IceSword 
from Xfocus.net. RootkitRevealer uses the same cross- 
view diff technique described in our previous paper on 
Inside-the-box GhostBuster [WBV+05]: it performs 
high- and low-level scans and reports the differences 
between these scans. For the file scans, it performs 
two low-level scans by reading the NTFS Master File 
Table and the NTFS on-disk directory index structures. 
For the low-level Registry scan, the tool reads the raw 
Registry hive files. A file discrepancy is reported if a 
file does not appear in all three scans. A Registry dis- 
crepancy is reported if the data, length, or type of a 
Registry value differs or if an entry is missing. 

Blacklight is designed to detect hidden processes 
and files via a kernel-mode driver. In addition to run- 
ning as a standalone process, it incorporates detection 
evasion technology: it may perform its scans through 
the Windows Explorer process. IceSword uses kernel- 
mode technology to detect hidden processes, hidden 
ports, hidden services, hidden auto-start programs, 
hidden files, hidden Browser Helper Objects, and hid- 
den Registry entries. It monitors process creation and 
deletion and is able to disable filter drivers that pre- 
vent file creation and deletion. In order to achieve its 
functionality, the program loads a kernel-mode driver 
and then disables kernel-level debugging. Unlike the 
other rootkit tools, it has anti-rootkit attack technology 
to keep it from being disabled by malicious software. 
It achieves this by trapping keyboard strokes and 
requiring that the user hit Ctrl+Alt+D in order to put 
the program into a mode where it may be shutdown. 


Similar rootkit problems exist on the 
Linux/UNIX platforms as well [PFM+04, YKS, YC, 
YW98, B99, YAO3]. (In fact, the term “rootkit” origi- 
nated from the root privilege concept on UNIX plat- 
forms.) A common technique used by Linux/UNIX 
rootkits to hide resources is to intercept system calls to 
the kernel via a Loadable Kernel Module (LKM) [ZK, 
YJ, JO1]. For example, some rootkits are known to hook 
read, write, close, and the getdents (get directory entries) 
system calls. More advanced rootkits can directly patch 
the kernel in memory [YC98, YLO1]. We discussed 
cross-view diff-based hidden resource detection for 
Linux/UNIX platforms in our previous — paper 
[WBV+05]. 


As a final note, most of today’s Windows rootk- 
its do not modify OS files or memory image; rather, 
they “extend” the OS through ASEP hooking in a 
way that is indistinguishable from many other good 
software programs that also extend the OS. Therefore, 
it is difficult to apply the genuinity tests and software- 
based attestation techniques that detect deviations 
from a known-good hash of a well-defined OS mem- 
ory range [KJ03, SPDK04]. On the other hand, these 
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techniques can detect both hiding and non-hiding mal- 
ware programs that modify the OS and are comple- 
mentary to the cross-view diff approach. 


Summary 


User-mode rootkits are popular because they are 
more portable and reliable than kernel-mode rootkits. 
We have shown that there is a quick and easy way to 
detect all user-mode rootkits: by performing a cross- 
view diff between a high-level infected scan above 
rootkit interception and a low-level clean scan below 
the interception, our tool can precisely detect hidden 
Registry entries and processes within a few seconds. 
The simplicity and efficiency make it an attractive tool 
for scalable deployment in large enterprises to provide 
protection against new or customized rootkits that 
escape common signature-based anti-malware scan- 
ning. Detection results from actual deployment sug- 
gested that hiding Trojans, most likely installed 
through malicious Web sites, may be an even more 
serious concern than rootkits in terms of prevalence. 
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ABSTRACT 


Since the LISA conferences began, the character of a typical “large installation” has 
changed greatly. Most large sites tended to consist of a comparatively small number of hand- 
crafted “‘servers” supporting a larger number of very similar “clients” (which would usually be 
configured with the aid of some automatic tool). A modern large site involves a more complex 
mesh of services, often with demanding requirements for completely automatic reconfiguration of 
entire services to provide fault-tolerance. As these changes have happened however, the tools 
available to provide configuration management for a site have not evolved to keep pace with these 
new challenges. This paper looks at some of the reasons why configuration tools have failed to 
move forward, and presents some suggestions for enabling the state of the art to advance. 


Background and Motivation 


Configuration Tools have been an important 
theme at LISA for many years, and most conferences 
include one or more papers in this area. Despite 
increasing recognition of the importance of the config- 
uration problem, there remains both a lack of concep- 
tual commonality and a lack of progressive innovation 
in the area. 


No significant standards have so far emerged, and 
new tools often merely espouse variations on existing 
approaches, frequently using both completely new 
specifications and code. For example, ten years sepa- 
rate the initial description of LCFG [5] and the presen- 
tation of Newfig [12], yet there is no clear evidence of 
conceptual progress. There is little or no attempt to 
create standards in a way which would reduce the bar- 
rier to new development or enable a greater shared 
understanding of the configuration problem. 


Within the configuration management commu- 
nity, a great deal of ongoing discussion revolves around 
this apparent failure to either successfully disseminate 
the concepts underpinning existing tools, and the expe- 
riences gained from those tools, or to realise them in 
tool implementations suitable for a wider audience. 


Furthermore, despite the slow rate of progress, 
no sign of convergence between tools is apparent. 
There are essentially no standards in this area, with 
each tool being not only entirely coded from scratch, 
but also unable to interoperate with other tools, or 
share configuration data with them. Although the Con- 
figuration Description, Deployment and Lifecycle 
Management (CDDLM) working group of the Global 
Grid Forum (GGF) has published a standard interface 
for configuration tools on grid fabrics, their focus is 
upon exposing a web-service interface to the underly- 
ing configuration system, which falls somewhat wide 
of the mark in addressing the current problems faced 
by system administrators. 


This absence of standard tools has lead to many 
large sites developing their own tools. To create a 


functional configuration management system in its 
entirety, however, is a significant undertaking, and 
even those sites which have been able to invest suffi- 
cient resources in a system to make it sustainable have 
been unable to gain a wider community of users. It 
seems likely that in many cases, the sheer difficulty of 
developing and maintaining a large monolithic system 
(often written by people with a shortage of time and 
no background in software development) has limited 
both the functionality and the portability of tools. 


It is surprising indeed, given the importance of 
this area, that we find ourselves typically unable to 
recommend any current system to interested users. 


Levels of Configuration 


Perhaps because of the above difficulties, most 
existing tools deal with configuration specifications at 
a very primitive level; they involve “files” and “per- 
missions”, rather than “high-level” concepts such as 
“services”, and the relationships between nodes — 
Figure | shows some typical statements at increas- 
ingly higher levels of abstraction. 


Ultimately, the requirements for a service are 
always expressed in high-level terms and most tools 
require these to be manually translated first into some 
lower-level requirements. As systems become more 
complex, this process is an increasing source of errors. 
Manual intervention is also unacceptable for auto- 
nomic systems which must reconfigure automatically 
in response to demand and failure. A future generation 
of configuration tools will need to be able to accept 
high-level requirement specifications, and to reason in 
much more sophisticated ways in order to determine 
the appropriate details. This is an inevitable conse- 
quence of the growing complexity of the modern 
installation, and the fact that the number of adminis- 
trators a site needs does not scale linearly with the 
number of machines managed. 


There is an analogy with computer programming 
here: early languages provided very little abstraction 
from the underlying machine, but as the size and 
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complexity of systems has grown, so has the separa- 
tion from the hardware operations. In a similar way, 
configuration tools must adapt to aid administrators to 
manage the complexity of their systems. 


Low Level 
“Copy this disk image onto these machines” 


‘*Put these files on these machines” 
“Put this line in sendmail.cf on this machine” 
“Configure machine X as a mail server”’ 


“Configure machine X as a mail server for this 
cluster” (and the clients will automatically 
be configured to match) 


“Configure any suitable machine as a mail server 
for this cluster’’ (and the clients will 
automatically be configured to match) 


Configure enough mail servers to guarantee an 
SMTP response time of X seconds 
High Level 


Figure 1: Levels of configuration specification. 


The Configuration Tool Zoo 


There are a plethora of tools available which pur- 
port to address some aspect of the configuration man- 
agement problem. Many of these are described in 
more detail in surveys such as [6]. However, we 
present the following examples as typical, in that they 
demonstrate a variety of the most common 
approaches. In particular we note that: 

¢ The tools have different /exicons — for example, 
there is no common agreement on how to spec- 
ify the “mail relay” for a particular client. 

e They use different languages and file formats. 

¢ Their deployment operations provide different 
guarantees regarding pre- and post-conditions. 


In general, these tools are also intended for direct 
manual use and present no intermediate interfaces 
between the human-generated specification, and the 
direct manipulation of the target machine. 


Arusha 


Arusha [11] uses an XML-based source language 
for describing configuration information. It is intended 
to facilitate sharing of configuration information 
between sites. 


LCFG 


LCFG [5, 7, 2] has a site-wide repository of declara- 
tive configuration data which is compiled into XML 
descriptions for transmission to the individual hosts. 
Clients execute components, each of which takes its 
specification from a corresponding section of the 
XML profile. 
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Quattor 


Quattor [3] has a similar architecture to LCFG, 
differing principally in the exact forms of the compo- 
nents and the capabilities of the central compiler. Both 
Quattor and LCFG have been used successfully to 
specify ‘‘complete” configurations for large, complex 
sites. 


BCFG 


BCFG [9] is a new tool from Argonne National 
Laboratories, based upon a highly centralised model 
of configuration construction. Complete configuration 
file sets are computed centrally for each client 
(together with attribute data), before being distributed 
in an XML envelope. 


Cfengine 


Cfengine [8] is probably the most widely-used 
“configuration tool,” but perhaps the most difficult to 
fit into a common model. It is frequently used to spec- 
ify only partial configuration information, and _ its 
specification language is less well suited to manipula- 
tion by independent programs. 


The CDDLM Standard 


The CDDLM standard proposal [1] is interesting 
in that it represents the first formal attempt at a stan- 
dard in this area that we are aware of. It is a proposed 
international standard which provides a tool-indepen- 
dent specification for the transmission of configura- 
tion information. The intention is to provide complete 
configuration control, aimed as it is at the manage- 
ment of grids — a highly federated environment requir- 
ing autonomic operation. 


A Common Model 


Despite the range of views and models set out 
earlier, there are in fact two distinct capabilities a 
modern system must provide: 

1. Deciding what configuration a particular node 
should have. 
2. Creating the desired configuration on a node. 


This is the difference between, for example, 
deciding that a machine must have a package installed 
on it, and actually installing that package. The funda- 
mental idea is to separate these two distinct roles into 
separate systems: systems that communicate through a 
configuration description. For the sake of brevity, in 
future we will call a tool able to decide what the con- 
figuration of a node should be a configuration man- 
agement system, and a tool capable of creating a con- 
figuration on a node a deployment engine. 

Writing a deployment engine requires an enor- 
mous amount of code, and keeping it current requires 
continuous maintenance. The systems we are asked to 
manage were not typically designed for neither bulk 
nor automated maintenance. There are a myriad of file 
formats, a slew of ways to restart a service, endless 
configuration files and preconditions. Each of these 
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will typically find a place somewhere within the 
deployment engine. 


At present, each configuration tool author begins 
by writing a deployment engine, and many deploy- 
ment engines exist (albeit inside a configuration tool, 
and not accessible independently). Once the deploy- 
ment engine is complete, whatever time and resources 
remain can be devoted to configuration management. 
Unfortunately, as we mentioned earlier, the amount of 
time and resources remaining tends to be small, and 
configuration management has remained in a rela- 
tively primitive state for many years. At present, the 
only mechanism typically available is the ability to 
group nodes and configure them as a unit. 


An overview of some existing configuration man- 
agement operations is given in “Managing Configura- 
tion Data” together with a discussion of proposed 
alternatives. We argue that the core of the difficulty in 
advancing the state of the art is that each current 
project must include both a deployment engine and a 
configuration management tool. Whilst deployment 
engines are well-understood and well-developed, they 
represent such a large amount of work that must be 
performed before configuration management can begin 
that no time is available to develop advanced configu- 
ration management systems. This is despite the fact 
that it is not our inability to deploy a configuration on a 
node that makes a large modern installation so hard to 
manage, but that the amount of information involved is 
too large to be managed in an ad-hoc fashion. 


By separating out the deployment engine from 
the configuration management tool, we make it possi- 
ble to implement one without implementing the other. 
We believe that this could lead to a rapid advance in 
the sophistication of management tools. In addition, 
we believe that many of the existing configuration 
tools discussed previously could be released as 
deployment engines relatively easily, as their configu- 
ration management functions are both small and 
peripheral, so one might imagine easy to isolate. This 
is certainly true for LCFG, for example. 


A Document Interface to Deployment Engines 


In this paper we aim to convince the reader that if 
deployment engines conformed to a minimal configu- 
ration description standard, it would allow the indepen- 
dent development of configuration management tools. 
This appears the only plausible way to relieve the cur- 
rent stagnation in configuration tool development, and 
enable the creation of a new generation of configura- 
tion tool, capable of managing a modern network. 


This section presents one possible interface to 
deployment engines, which we believe is most use- 
fully realised as a document. This is similar to the 
familiar tool chain for program development (e-.g., 
autoconf, automake, make, gcc, Id, etc.) More impor- 
tant than the format of the document is what such a 
document represents: 
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A configuration description specifies a complete, 
unambiguous and instantaneous configuration for the 
target machine or system. 
© Complete means that everything the deployment 
engine is capable of specifying is given a value. 
¢ Unambiguous means that no further logic is 
required to determine the parameters of the 
deployment engine. There are no references, no 
late binding, nor any database queries required. 
¢ /nstantaneous means that the specification is a 
snapshot of a node’s state that the engine 
should be working towards. There is no time 
dependency stated. 


Our proposed document format is XML-based, 
simply on the grounds that XML is a widely-used and 
convenient way of representing structured data such as 
configuration parameters. Most languages provide 
XML parsers and processing libraries which simplify 
the development of cross-platform libraries and tools. 
There is no suggestion here that tools should provide 
users with an XML interface, only that it is a conve- 
nient format for exchanging data between a manage- 
ment tool and a deployment engine. 

Entries 
<entry name="foo"> 
<value>bar</value> 
<origin> 
<file name="foo.cfg" line="132"/> 
</origin> 
</entry> 
<entry> 
<value>baz</value> 
<origin> 
<file name="anon.cfg" line="211"/> 
</origin> 
</entry> 


An entry represents a data item. An entry may be 
either named or unnamed. When named, the entry rep- 
resents a key-value pair, the most common method of 
storage for configuration data (e.g., [10, 5, 9, 3]). The 
entry’s name attribute is the key, and the value is 
inside the value element (which must be text — there 
may not be any nested elements.) In the example, the 
key “foo” was bound to the value “bar”. An entry 
may also be unnamed, for example for use as a mem- 
ber of a list of values. The origin element is optional, 
specifying a list of files and lines which were used to 
determine this value. It is purely to enable a deploy- 
ment engine to give meaningful errors when the files 
the user edited were passed to a different tool. 


Structs 


<struct name="sudoers"> 
<entry> 
<value>joe</value> 
</entry> 

<entry name="admin"> 
<value>jane</value> 
</entry> 

</struct> 
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Like entries, structs have an optional name. They 
provide a means of structuring data items. Structs may 
contain both named and unnamed entries and structs. 
A struct containing only named entries can be thought 
of as a record, a struct containing only unnamed 
entries is a list. The elements contained in a struct are 
ordered. Structs may not contain character data (only 
structs and entries.) 

Configurations 
<configuration target="lcfg-engine" 
version="1.0.3"> 
<struct name="applications"> 
<entry> 
<value>emacs</value> 
</entry> 
</struct> 
</configuration> 


The configuration element is the root of the 
XML document. It serves to contain all the other ele- 
ments. The target attribute indicates the deployment 
engine for which this configuration is intended, and 
the version attribute a minimum version number for 
that engine. This is largely to provide sanity checking 
(you wouldn’t want to accidentally hand a configura- 
tion to an old version of the tool and have it trash a 
machine in confusion.) 


Avoiding the Phantom Lexicon 


This section has presented one possible approach 
to defining a “low level” configuration description 
document format. We have sidestepped the issue of a 
standard lexicon (that is, defining a standard set of 
keys whose values should be similarly interpreted by 
all compliant deployment engines.) We argue that such 
a standard lexicon could not be meaningfully defined 
in the present environment, where there is so little 
commonality between existing tools. 


Our purpose is to enable progress towards greater 
sophistication in configuration management by provid- 
ing a standard method of interfacing with low level 
deployment engines. Whilst a standard lexicon would 
both allow sites using different deployment engines to 
share configuration data, and allow them to migrate 
easily between engines, we do not believe there is 
enough understanding of what the appropriate lexicon 
would be to define it at present. The most pressing 
need in this area is to begin to decompose the problem. 
When the issues involved are better understood, it will 
be possible to standardise a lexicon. If we cannot start 
moving forward, we will never reach that point. 


The choice of the “level” at which to specify 
this interface is crucial; if the level is too low (it con- 
tains too little structuring information), then it will not 
be possible for common tools to perform meaningful 
operations on the configuration. If the level is too 
high, then the mismatch between the common format 
and the assumptions of existing tools will be too great, 
and no meaningful translation will be possible. This 
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proposal is based on a study of the existing tools in the 
previous section (and others), and we believe that an 
interface at the level presented is the minimum required 
to support useful common operations (see the next sec- 
tion), while providing a large degree of architectural 
freedom for the tools themselves. 


Managing Configuration Data 


It might seem that the interface set out earlier is 
too limited to support sophisticated management tech- 
niques. In this section, we will attempt to show that a 
wide range of transformations can be applied to struc- 
tured configuration data, without needing to have the 
management tool “understand” that data. There is an 
analogy here with the use of simple UNIX command 
line tools, applied in a filter chain, to perform more 
complex tasks: tools such as sort, grep, m4, and even 
awk can be combined to perform powerful transforma- 
tions on text files, even though the programs them- 
selves have very little knowledge of the text file struc- 
ture apart for a simple division into fields and records. 


Classing 


The most widespread operation available in con- 
figuration tools to manage configuration data is class- 
ing: groups of machines are assigned to particular 
classes, with configuration data being associated with 
each class. Alternatively, this can be seen as establish- 
ing predicates about machines, and associating with 
each predicate a set of configuration information. 


A simple example might involve a “webserver” 
class, associated with configuration data that specifies 
webserver specific packages and configuration options. 
In order to deploy a new webserver, it is only necessary 
to place it in the “webserver” class, and all the neces- 
sary configuration files and packages will be deployed. 


Basic classing gives the administrator an enor- 
mous increase in the manageability of their network. 
By separating out roles into classes, it becomes possi- 
ble for an administrator to create complex combina- 
tion configurations very quickly. Complex updates can 
also be rapidly deployed, as changes can be made to 
class definitions rather than to individual machines: if 
a bug is found in xinet on Fedora Core 3, it suffices to 
update the class definition for “Fedora Core 3 
machine,” which all FC3 machines inherit. There is 
no possibility of missing, or overlooking a machine, 
nor is it necessary to modify each machine by hand. 


Advanced Classing Operations 


Whilst almost all configuration tools provide 
their own mechanism for managing simple classes of 
objects, this feature still remains to be fully explored. 
The simple scenarios described above are both power- 
ful and useful, but as the use of automation matures, 
administrators find there are unanswered questions in 
this approach. Consider a “highly-secured” class, and 
a ‘“‘web-server” class, each managed by different 
teams. These classes have different objectives, and 
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overlapping domains. It is entirely possible specifica- 
tions from one class will conflict with another. To our 
knowledge, no current configuration tool presents a 
satisfactory answer to the question of how conflicts 
can be resolved. 


Current suggestions for progress in this area 
include: 
¢ Prioritisation A machine might belong to dif- 
ferent classes with different priorities, enabling 
the data specified by one class to be out- 
weighed by another. Or classes might set values 
with different priorities, so allowing them to be 
safely overridden elsewhere. This would avoid 
the order-dependent (or even oscillatory) deci- 
sion process displayed by current tools. 
¢ Constraints Rather than specifying definite 
values for properties, it might be possible to 
specify ranges of values. This would enable a 
tool to mediate between different groups auto- 
matically. Consider the web-server team speci- 
fying that either port 80 or 8080 be used, and 
the security team that only ports over 1000 be 
used. These requirements are not conflicting, 
but had the webserver team simply said port 80, 
there would be no way for the tool to mediate. 
Aggregation 

It is common for some aspect of the configura- 
tion of a system to be derived from the configurations 
of other machines. This might be a client being con- 
figured with respect to its servers, or a server from its 
clients. Specifying this information manually is error 
prone and time consuming. By introducing automa- 
tion, we both save ourselves time and increase our 
confidence in the correctness of our network’s config- 
uration. Typical examples include: 

e A firewall host may want to open holes for ser- 
vices marked as public. Specifying this in the 
configuration of the hosts, rather than in the 
configuration of the firewall, means that when 
the host’s service is decommissioned, so is the 
firewall hole (automatically). 

e A DHCP server might want to allocate fixed 
addresses to individual nodes. Again, the fixed 
address for a host is most naturally stored with 
the host itself. This means the server must 
aggregate those values into its configuration. 
This can be done systematically by a manage- 
ment tool. 


The only tool we are aware of that provides explicit 
aggregation operations is LCFG, although Quattor and 
others have the ability to derive parts of a host’s configu- 
ration from a database. Future research in this area is 
likely to focus upon the ways in which aggregation can 
be provided without a central configuration server — it 
has been suggested that one way to achieve this would 
be through the use of peer-to-peer technologies. 


Again we note that, given just a structured tree of 
data, it is possible to implement a generic aggregation 
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function, for example by converting values across a 
group of files into a list of values in the target file. 


Sequencing and Planning 


Configurations cannot be changed atomically, as 
there are frequently multiple dependents upon a single 
value. For example, it may be necessary to both bring 
up a new server and redirect all clients to point at it, 
before the old server can be taken down. Thus far we 
have avoided talking about how configuration changes 
can be sequenced. To the best of our knowledge, none 
of the available tools tackle this question. 


One possible approach, as yet untested, is to allow 
the user to specify a set of invariants that must remain 
true during the deployment of any configuration. These 
can be used to break down a deployment into multiple 
steps, each of which maintains the invariants. One 
example of an invariant might be “A client must not be 
configured to point at a non-existent server.” 


Although there can be significant sophistication 
required to maintain an arbitrary invariant, simple 
invariants of the kind given above can be shown to 
generate safe 3-step transitions between configura- 
tions. Of course, some integration with monitoring, or 
user-feedback, would be needed in a real tool to step 
through the intermediate configurations. It does little 
good to create several intermediate steps then deploy 
them all at once! 


The important point here is that the specification 
of invariants can be done in a general way (all that is 
required is to be able to establish requirements on 
pairs of values — to the tool it is unimportant what 
those values represent.) This is an area in which even 
very simple tools can generate large advances in the 
state of the art. 


Delegation and Authentication 


A large installation is not managed by a single 
person. Often there will be several teams working 
together, each tasked with maintaining different 
aspects of the network’s configuration. We discussed 
the possibility of using a configuration management 
system to perform some automatic mediation between 
teams. Here we are concerned with the security issues 
involved in delegating configuration aspects to people 
outside of the central teams. 


Configuration management systems at present 
have a boolean notion of authorisation: a user is either 
authorised to perform any action whatsoever on any 
machine, or they are not authorised to perform any. 
Although we might feel that there are many people 
involved in the management of a system who should 
have some limited control of some aspects of a 
machine, current tools do not enable us to act upon 
that idea. 


The simplest example is allowing users to con- 
trol their own machines to some extent. It seems that 
many sites allow users a choice either of complete 
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freedom and responsibility, or of accepting a com- 
pletely prescribed system. Machines may have giga- 
bytes of unnecessary software installed on them 
because users are unable to choose which software is 
most applicable to them. 


The solution to these problems is to introduce a 
notion of limited delegation, or authorisation, into our 
tools. For example a constraint-based system could set 
hard boundaries on what a user can do to their 
machine, while still providing much more flexibility 
than is currently available. Nor would it be necessary 
for users to understand the tools if unprivileged user- 
space helpers were available to guide them through 
the options. 


Again, we believe that authorisation and limited 
delegation are features that could be explored without 
a known underlying lexicon to work to. Restrictions 
and security information can be identified with areas 
of the configuration data tree, and are no less securely 
or usefully enforced for not being understood by the 
tool that does so. 


An Example 


The following example demonstrates how some 
of the above features might be used to implement the 
comparatively high-level configuration requirement: 


Configure two DHCP servers on every Ethernet 
segment. The DHCP servers should provide 
fixed IP addresses for all the other machines on 
the segment. 


Conventionally, this policy might typically be imple- 
mented using a partially automated approach such as 
the following: 

e Two appropriate machines on each ethernet 
segment might be identified manually. 

e The lists of machines on each segment, together 
with their corresponding IP and MAC addresses 
may be extracted from some database using an 
ad-hoc script and massaged into the right form 
for the configuration files. 

¢ The configuration files would be distributed to 
the appropriate machines, and the DHCP dae- 
mons started on them, possibly with the aid of 
some tool such as cfengine or LCFG. 


Of course, there are many others ways of doing 
this, but this is certainly typical, and this particular 
example provides a simple illustration of several 
important configuration manipulations. 


Note that most existing ‘“‘configuration tools” 
would be able to handle the last operation automati- 
cally, but probably not the first. This illustrates the dif- 
ferent between configuration deployment and configu- 
ration management. The manual operations involved in 
the above configuration management process make it 
unsuitable for an autonomic environment; if a DHCP 
servers fails, a new one must be selected manually. The 
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disconnect between the independently created IP data- 
base, and the actual deployed machines is also a poten- 
tial source of errors (if we decommission a machine, 
can we guarantee that someone will remember to 
remove the MAC address from the database’). 


A configuration management tool must accept 
the high-level statement of the requirement, and trans- 
late it automatically into a form suitable for the 
deployment engine. For example, this might involve 
the following process: 

¢ A monitoring system would provide a list of 
active machines on each ethernet segment. 

¢ A constraint process would be used to select 
two (active) candidate machines on each seg- 
ment. 

¢ A classing mechanism would allocate a DHCP- 
server class to each of these machines. This 
class would define the appropriate parameters 
to configure and start the DHCP service. 

e An aggregation mechanism would collect the 
IP/MAC mappings from the individual node 
configurations and make them available as part 
of the server configuration. 


The result of this process would be a complete, 
unambiguous configuration specification for each 
machine which could then be passed to the deploy- 
ment engine. This solves the two problems of the 
manual approach noted above; replacements will auto- 
matically be configured for failed servers, and 
machines which are decommissioned will automati- 
cally be removed from the DHCP configuration files. 


The result of the configuration management 
process is clearly expressible using the simple docu- 
ment interface outlined earlier. Clients would be con- 
figured to run DHCP client software (with no special 
parameters), and servers would be configured to run 
DHCP servers (with a specified list of IP/MAC 
address pairs). The deployment engine would need to 
translate these simple requirements from the XML 
description into the appropriate configuration files and 
daemon operations — a conceptually simple process, 
but one which may involve a considerable amount of 
code to handle the details. 


It is important to note that most of the above 
configuration management operations themselves are 
quite generic; there are many other applications of the 
operations such as “‘classing’ and ‘“‘aggregation” — 
they are not specific to the DHCP example, and can be 
implemented as generic operations on the simple 
XML structures proposed earlier. 


Conclusions 


Advances in the sophistication of modern config- 
uration tools have been hampered by the inability to 
decompose the problem into subtasks with well defined 
interfaces. If everyone who wanted to create a program- 
ming language chose their character set differently, 
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wrote their own editor, then created the entire tool- 
chain, we would not see the sophistication and flexi- 
bility that we do today. 

This paper proposes a simple level of interopera- 
tion which should allow independent development of 
higher-level configuration management tools, without 
the need to reinvent the wheel at each step. Such tools 
would benefit from the full range of available deploy- 
ment engines, and separate out the maintenance bur- 
den onto more than one team. 


We are sufficiently realistic to recognise that 
many tools, developed for internal use, will not be 
modified simply to meet the requirements of some 
external standard unless there is a significant practical 
benefit. We do hope, however, that the principles out- 
lined herein will at least influence the development of 
new configuration tools, and promote a higher degree 
of interoperability in the future. 


We would like to invite comment and discussion 
of the issues raised here on the Issconf mailing list’. 
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ABSTRACT 


While configuration management systems are generally regarded as useful, their deployment 
process is not well understood or documented. In this paper, we present a case study in configuration 
management tool deployment. We describe the motivating factors and both the technical considerations 
and the social issues involved in this process. Our discussion includes an analysis of the overall 
effect on the system management model and the tasks performed by administrators. 


Introduction 


System administration is, at its heart, the profes- 
sion of helping people to use computers. System 
administrators are domain experts who provide 
impedance matching between users’ desires and com- 
puters. The expertise of system administrators is mani- 
fested in their choices of computer hardware and soft- 
ware and of system configuration. Environments are 
constantly changing; the need for timely software 
updates and frequent configuration changes has never 
been greater. Configuration management tools provide 
different levels of automation and representational 
models, but all have the same basic goal: to help in the 
system configuration process. 


Nevertheless, adoption of configuration manage- 
ment systems has lagged substantially behind tool 
development. The explanation rests largely with the 
up-front cost of building an adequate representation of 
an environment, but the problem can also be traced to 
the requirement that administrators change their 
administration methods. This change makes the con- 
figuration management adoption process quite costly 
and time consuming. 


The Mathematics and Computer Science Divi- 
sion of Argonne National Laboratory consists of 
nearly 200 researchers, programmers, students, and 
visitors. The division is home to several hundred 
workstations, three large clusters, and other high-per- 
formance computing resources. One group maintains 
this diverse set of resources. Recognizing the need for 
more efficient management mechanisms, members of 
the system staff have contributed to a number of sys- 
tem management research efforts [6, 4, 7]. 


In the summer of 2004, the group decided to 
deploy Befg2 [3], a configuration management tool 
developed in house. As of May 2005, much of the 
deployment has been completed. Bcefg2 manages the 
general infrastructure and two clusters and is being 
deployed on another cluster and on an IBM Blue 


Gene/L system. The general infrastructure is extremely 
complex and involves the largest number of adminis- 
trators, hence; we focus on this particular deployment. 


This paper documents our experiences and 
lessons learned in adopting a configuration manage- 
ment tool. We discuss the goals that prompted this 
adoption and describe the deployment process in terms 
of both technical and — more important — social issues. 
The adoption of Bcefg2 has dramatically changed the 
procedures and model used by system administrators 
in the division. We discuss these outcomes in detail. 


One cannot talk about configuration management 
without including technical aspects that are likely tool- 
specific. As the same time, we believe that many of 
the issues we faced and benefits we reaped in deploy- 
ing Befg2 are intrinsic to the adoption of any configu- 
ration management system. Where possible, we avoid 
the discussion of Bcefg2-specific topics. 


Since configuration management has been recog- 
nized as a problem for quite some time [4, 9], numerous 
tools have been written to address this task. Many of the 
tools, in particular LCFG [1] and Quattor [8], can cause 
large changes in management methods; our discussion 
of Befg2 [3] is especially applicable to such tools. 
Other tools, such as SystemImager [5] and CFEngine 
[2], have a model that is not substantially different from 
manual system administration; for these tools our dis- 
cussion of deployment issues is less applicable. 


Architecture 


In this section, we describe the overall goals that 
motivated our adoption of Bcfg2. We also briefly 
describe Bcfg2, in order to provide an understanding 
of the outcomes we achieved. 


Goals 
The bulk of the benefits we hoped to achieve 


with better configuration management were efficiency 
related. We were spending an inordinate amount of 
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time performing software upgrades and applying secu- 
rity patches. In partial response to this problem, we 
found ourselves faced with the deployment of a new 
base operating system, Debian Sarge. This occasion 
provided an ideal opportunity to add a new configura- 
tion management system into the environment. 


We wanted a centralized location where all con- 
figuration information for all clients could be stored 
and coordinated. This requirement raised a variety of 
expressivity issues. We wanted all aspects of the con- 
figuration specification to be as terse as possible. That 
is, we wanted to express all required configurations in 
a way that minimized redundancy in the specification. 
Similarly, we wanted configuration changes to be made 
in as simple a way as possible. For example, adding a 
software upgrade to all systems or uniformly changing 
the contents of a configuration file should be trivial. 


We also wanted a high-level interface into the 
configuration management that allowed configuration 
specification based on machine role. For example, it 
should be easy to ask for another instance of a given 
role, like a web server. 


Another important goal was to eliminate config- 
uration state local to clients. That is, all machines 
should match their configuration specification. Once 
no local configuration exists, no local configuration 
data will be lost in the event of a machine rebuild. 
Adherence to this goal enables machines to be trivially 
rebuilt. Furthermore, once the configuration specifica- 
tion contains all configuration directives needed to 
produce a goal, this goal can be replicated. 


Finally, we wanted good practices encoded in the 
configuration management tool. The tool should per- 
form operations in the safest manner possible. It 
should also make every attempt to ensure that new 
configuration specifications are activated. 


Befg2 Overview 


While the Befg2 architecture is not the main 
focus of this paper, several details are required to 
understand our deployment. Bcefg2 has a client-server 
architecture. The server is responsible for building 
configurations for clients, based on a global configura- 
tion specification. This specification contains high- 
level directives for clients, referred to as the metadata, 
and a set of configuration rules that can be used with 
the metadata to produce literal client configurations. 
That is, they contain information that needs no further 
processing for client use. These configurations are 
also assumed to be comprehensive; they contain infor- 
mation for all aspects of client configuration. For 
example, a software package or service that is active 
on a client will be included in its configuration. Any 
installed configuration entities not listed in the config- 
uration are flagged as “extra.” The Bcfg2 client uses 
heuristics to discover this “extra” configuration. 


The Befg2 client connects to the server and 
downloads its configuration. It then inventories the 
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local system and compares this inventory to the con- 
figuration specification. Any discrepancies found in 
this process are flagged for later correction. Next, the 
Befg2 client runs its heuristics to find extra configura- 
tion. Anything located in this pass is similarly flagged 
for later correction. After the detection work is done, 
the Bcfg2 client rectifies any conditions previously 
found. This behavior is tunable; dry-run and extra con- 
figuration removal modes are available. 


Once the client has completed operations, it 
uploads statistics to the server. The information 
includes the overall machine state (clean or dirty) and 
lists of the failing and modified configuration entries. 
This information is stored on the server and can be 
used to generate nightly reports about the overall state 
of the network and its correspondence to the configu- 
ration specification. 


Deployment 


Deploying Befg2 in our environment took a sub- 
stantial amount of time. Our experiences since its 
deployment, however, have more than justified this 
investment. In this section we describe the technical 
and social issues involved in the process. Following 
this, we discuss various improvements to the adminis- 
trative process enabled by this deployment. 

Technical Deployment 

Deploying Bcfg2 took approximately four 
months of work performed primarily by one person. 
As is frequently the case with systems like this, the 
first 90 percent of the work was completed in the first 
six weeks, while numerous small issues were resolved 
over the course of the next 10 weeks. 


Deployment was initially undertaken as part of a 
base OS upgrade, moving from Redhat 7.3 to Debian 
Sarge. We chose this occasion because our previous 
experience with a mid-stream switch to Bcfgl had 
proved problematic. Basically, systems managed in an 
ad hoc fashion tend to have a lot of configuration 
inconsistencies. Ensuring that these machines can be 
cleanly rebuilt by using a configuration management 
tool can be quite difficult. The introduction of a con- 
figuration management tool is possible but must be 
carefully performed. In contrast, the deployment of 
new machines can be easily done, as the deployment 
process is available for examination. 


The first goal was to get an automated build sys- 
tem working properly. Our environment has two main 
types of machines: workstations and servers. We 
decided to use SystemImager [5] for initial client 
installations, calling Befg2 before initial reboot. This 
approach ensures that machines come up properly 
configured and secure upon first reboot. Two profiles 
for Befg2 were created and made selectable from the 
initial SystemImager boot menu. 


Creating the configuration profiles for workstations 
and servers was not difficult. This process consisted of 
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specifying all configuration aspects of workstations, 
including all environment-specific modifications. A 
workstation configuration had been created for Bcfg] 
and was easily migrated to Befg2. Had this not been 
the case, the creation of an initial configuration would 
have taken a few days. This process consists of 
recording all important aspects of configuration in the 
central specification. Aspects can be quickly incorpo- 
rated into the specification. Our workstation configu- 
ration initially consisted of nearly 1,100 configuration 
aspects. Once this process was completed, we con- 
structed a server configuration as a subset of the work- 
station configuration, since many of the configuration 
aspects of these classes are similar. 


Once a simple build mechanism was in place, we 
rebuilt administrator desktops and some test servers 
using the new image and management system. In one 
case, the administrator ran with two desktops concur- 
rently for several weeks, in order to find subtle aspects 
of needed workstation configurations not yet included 
in the workstation profile. The process paid a big divi- 
dend; by the time “‘normal” users started using work- 
stations based on the new build, few configuration 
problems remained. 


After the new system became full-featured and 
stable enough, we started to allow users to request 
machines with it. These early users provided the 
remainder of input regarding missing configuration 
from the new workstations. After several of these users 
had successfully used new-build machines for some 
time, we began upgrades for the rest of the division. 


In this stage the simplified build process was par- 
ticularly beneficial. Three people performed most of 
the workstation rebuilds. The initial target was to com- 
plete most machine upgrades in four weeks. Desktop 
rebuilds were a simple process. Each machine has a 
local scratch disk, which gets cleaned upon rebuild. 
Users save any needed data stored there. Once this 
data is saved, rebuilds can occur at any time and take 
30 minutes. All user interactions occur in the first 
three minutes of the process; the rest of the process 
can complete unattended. In the course of a month, 
nearly 80 desktop machines were upgraded. 


After completing the workstation upgrades, we 
shifted our focus to the server machines. While the 
server profile had been used to build new machines, 
many existing servers remained unmanaged by Bcfg2. 
As these servers were replaced, we encountered a whole 
new set of issues relating to tool deployment — issues 
that were generally related not to tool completeness but 
rather to the way configuration changes were propa- 
gated to machines. For example, certain machines were 
deemed too important to perform automatic changes. 


This realization necessitated a major change in 
our deployment strategy. Initially, all machines had 
called Befg2 each night, and all required changes were 
performed. On workstations, this model was good 
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enough; while these machines are important, they 
don’t cause congestive failures when problems occur. 
For servers, however, we needed to run Bcfg2 in dry- 
run mode each night and send its state to the relevant 
administrator. While examining this issue, we also 
realized that cluster nodes should run Befg2 in yet 
another way: between jobs, in order to prevent inter- 
ference with user calculations. 


These experiences shifted our focus from a tool- 
based one to an administrator experience-based one. 
Initially, we were quite concerned about tool correct- 
ness and completeness. As Bcfg2 proved itself and 
bugs were fixed, our confidence in its correctness 
greatly increased. Also, our workstation configuration 
proved to be as complicated as any other in our envi- 
ronment, so the configuration specification process for 
our servers was fairly simple. 


Our change in focus amounted to the realization 
that the tool client-side functionality was not sufficient. 
The tool must also provide enough information for 
administrators to make effective decisions as conveniently 
as possible. Moreover, it must supply configuration state 
information in a convenient way. From this point onward, 
nearly all development focused on an information presen- 
tation system to provide a sort of scoreboard for the entire 
network and its configuration state. 


Administrative Improvements 


The deployment culminated with the implemen- 
tation of a robust reporting infrastructure. This infra- 
structure provides periodic information about the cur- 
rent configuration state of all clients, the time of their 
last contact with Bcfg2, and a list of pending configu- 
ration changes. The information allows administrators 
to observe the logic employed by Bcfg2 during normal 
operations. This single factor had more impact on 
administrator trust in Befg2 than all others. 


The reporting system results in emails, Web 
pages, or RSS feeds that contain information about 
either specific hosts or the overall status of all Befg2 
clients. Administrators can subscribe to reports about 
particular clients of interest, or all systems, enabling the 
detection of two common problems: clients not receiv- 
ing configuration updates and clients unable to perform 
needed configuration updates. Administrators can also 
observe the operations taken by Bcfg2 over time. 


These reports create a picture of the entire net- 
work that gives discussions about configuration a basis 
in fact. The reports greatly improved our understanding 
of our systems, their configuration, and its modification 
patterns. More important, administrators grew to trust 
Befg2, and hence allow the remainder of our network, 
composed of our most important machines, to be rebuilt 
and managed by Bcfg2. 


Social Issues 


While the technical aspects of Befg2 deployment 
were complex, managing the social aspects of the 
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deployment was far more difficult. This process 
occurred in an ad hoc way in our group and could have 
been substantially improved had we initially known the 
related considerations. 


Adoption of any configuration management sys- 
tem is a stressful process. Configuration management 
tools affect the whole of the system management 
process. All administrators are required to adopt new 
processes for achieving the same tasks they already 
know how to accomplish. Since such changes can 
directly impact the services system administrators are 
expected to provide reliably, tensions can run high. 


Communication also becomes an issue, because 
of the variety of perspectives held by different admin- 
istrators. We found that three main factors motivated 
most of our disagreements during the deployment 
process: trust in the tool, a belief in the benefits pro- 
vided by the tool, and the perception of a complexity 
increase or decrease caused by the tool. 


Trust is certainly the most important of these fac- 
tors. If a tool remains untrusted by administrators, it 
will never be substantially used in their environment. 
The trust-earning process certainly varies from person 
to person, but we can discuss the issues we observed. 
In general, as users gain more experience with the con- 
figuration management tool, they begin to trust it 
more. Two aspects of trust are important. The first is 
that the tool can properly represent any configuration 
state that the administrators may desire. This problem 
is largely technical and is solved as the administrator 
gains experience and familiarity with the tool. The sec- 
ond aspect of trust is harder to earn. Administrators 
must trust that the tool will perform the specified 
changes appropriately. This sort of trust is earned only 
through a long period of experience running the tool 
and observing the resulting configuration changes. The 
process can be accelerated, however, through the expo- 
sure of tool decision information. If administrators can 
easily examine the decision process each time they run 
the tool, then their trust will grow more rapidly. 


The second factor that guides the adoption 
process is the perception of benefit. All administrators 
in our group were already overburdened with tasks, so 
expecting them to spend time learning something new 
was difficult. If a management tool didn’t present a 
clear case for rapid improvement in efficiency, learn- 
ing about it wouldn’t be given high priority — and 
rightly so. Lack of time to experiment with a tool 
clearly has a detrimental effect on overall trust in the 
tool. Hence, adoption can be greatly hampered simply 
because of a lack of information. This factor can be 
minimized, however, by providing improvements that 
quickly benefit all administrators. Easy-to-adopt solu- 
tions to common problems provide a good incentive to 
try out a tool. 


The third factor in the adoption of configuration 
management systems is the perception of complexity. 
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The complexity increase may be minor; but to users 
unfamiliar with particular tool, this added complexity 
will be unattractive because it will lead to decreased 
efficiency for some tasks. Over time, as the adminis- 
trators become more familiar with the tool, the added 
cost of this complexity is reduced. Once the deploy- 
ment is complete, complexity is compartmentalized, 
as administrators can focus on their task and ignore 
others. For example, an administrator responsible for 
web servers can focus on apache configuration without 
worrying about upgrading ssh. 


All of these factors are heavily interrelated. 
Throughout the deployment process, their influence 
was felt through each of the issues we encountered. 


Disagreements 


Throughout the process of tool adoption, each 
administrator in the group internalizes more information 
about the new tool, building an opinion about the tool’s 
value and potential use cases. Each administrator will 
trust the tool to a different extent and will have different 
ideas about the potential benefits to be gained and the 
complexity costs involved. The following discussion 
describes the different points of contention that arose in 
our group. Each of the major issues is described in the 
abstract, along with the factors that turned out to be 
important. While we believe that these issues are repre- 
sentative, the list is by no means comprehensive. People 
like to argue about all sorts of issues. 

¢ Buy-in. Initially, everyone must agree that a given 

tool should be used. This issue is largely influ- 
enced by the trust and comfort levels administra- 
tors have with a tool, not to mention its technical 
correctness. Learning how a tool works is essen- 
tial, but this process can require substantial 
amounts of time, especially in large groups. 

e Existing investments. A working environment 
typically has a sizable investment in technical 
methodologies. Generally, a set of utilities has 
been developed to automate particular tasks, and 
a large body of institutional knowledge has 
evolved around specific tools and methodologies. 
Moreover, the creation of these processes and 
tools usually implies an emotional investment. 
Overcoming these investments without alienating 
members of the group is difficult — but possible. 
Level of control. Any tool will have a funda- 
mental set of assumptions or functionality that 
guides its behavior. Even if the tool is reliable, 
it can be difficult to convince everyone that a 
change is beneficial — especially if the methods 
that a proposed new tool uses to implement a 
given task differ from those historically used. 
This issue can be overcome only with a large 
amount of empirical evidence that these meth- 
ods are equivalent. As when programmers 
made the initial switch to high-level languages, 
administrators are accustomed to making com- 
plex low-level changes to systems, rather than 
using high-level specifications of functionality. 
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Our goal is not to diminish these concerns in any 
way; in fact, all of these considerations are quite rea- 
sonable. In some ways, these concerns illustrate the 
core essence of system administration. That is, the 
adoption of a tool that will radically affect every 
aspect of system maintenance is not a decision that 
should be taken lightly. Administrators, after all, 
shoulder the brunt of system failures, misconfigura- 
tions, and software configuration problems. 


Rather, our intent is to document these concerns. 
We believe we have gained a deeper understanding of 
our requirements and also provided a comprehensive 
vetting process. Moreover, through this documenta- 
tion, we hope to aid other groups attempting the same 
sort of transition. 


The highest-level problem can be most easily 
summarized as “buy-in.” Once that problem is 
resolved, the tool deployment will achieve critical 
mass and no longer serves as a point of contention. 
Hindsight 

Despite the social issues we confronted, we man- 
aged to implement a configuration management sys- 
tem. We attribute our success to three factors. 

¢ Our group was predisposed to recognizing the 
value of configuration management. This atti- 
tude removed an important initial hurdle from 
the process. Had we simultaneously needed to 
convince the group of both the need and mech- 
anism for configuration management, the out- 
look would have been far worse. 

© One administrator was involved in both the 

Bcfg2 development activities and maintenance 

of the division infrastructure. Without his work 

as liaison, many arguments wouldn’t have car- 
ried nearly the weight that they did; adoption 
could have easily stalled. 

¢ Our group is quite amicable, and not particularly 
sentimental. These characteristics allowed easier 
discussion of contentious subjects and the re- 
placement of existing mechanisms and tools. 


Even with these factors, considerable persever- 
ance and evangelism were required. The outcome of 
this process was in doubt throughout much of the 
deployment process. At many points, administrators 
did not seem to find the model compelling enough and 
did not trust the tool. Fortunately, everything worked 
out well in the end. 


Recommendations 


While there is a great amount of social variance 
among groups of system administrators, we can make 
several recommendations based on our experiences. 

¢ The tool under consideration must have an 
advocate who is technically respected by the 
group. His role will be to assess the various 
tools and select one that seems best suited to 
the environment. He will also need to convince 
other members of the group that the chosen tool 
is the proper one. 
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¢ Administrator concerns should be addressed, 
not ignored. These concerns are generally 
based on experience and reflect potential tech- 
nical issues that could occur later. Once all con- 
cerns have been addressed, the group will more 
readily accept the tool into daily operations. 

¢ Tool advocacy will be most compelling when 
improvements mentioned are useful in the short 
or medium term. Long-term improvements, 
while desirable, do not often provide short-term 
motivation. Hence, long-term benefits secure a 
position on the “when we have time” list for 
tool deployment. 


Outcomes 


In spite of the difficulties described above, this 
project has succeeded beyond our expectations. We 
have several new capabilities we could not have pre- 
dicted even six months ago. All of these capabilities 
have resulted from three major shifts in architecture. 
First, we now have a centralized configuration specifi- 
cation and statistics that allow reasoning about the 
desired (and actual) configuration states of our entire 
environment. Second, we now have an abstraction bar- 
rier between our specifications of machine function 
and the implementation of that function. This barrier 
simplifies both parts of the configuration specification 
and allows easier interactions. Third, several opera- 
tions have been completely implemented by the con- 
figuration management system, thereby reducing the 
cost in time, and improving the economies of many 
processes. Each of these improvements contribute to a 
fundamental alteration of the management model for 
our environment. 


Configuration Specification 


Many configuration management tools have a 
centralized specification that describes the desired 
state of the network. Where Befg2 departs from this 
model is the addition of detailed statistics about client 
state. The addition of these statistics to the central 
specification causes what would be a static set of rules 
to become a living document, automatically updated 
by Bcfg2. The desired configuration and all deviations 
from it are available in a central location. 


The existence of a configuration oracle for an 
entire network alters the administration mind-set in 
that global notions of state can now be constructed. 
Many data mining techniques can now be utilized. 
Reports describing network configuration _ state, 
reporting on the frequency and success of client con- 
figuration processes can give a thorough picture of 
unexpected client states before users are affected. Sim- 
ilarly, reports automatically generated from the config- 
uration specification can provide insight into current 
software revisions, client functionality, and potentially 
even system interdependence. Reports such as these 
can prove useful for auditing purposes, the training of 
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new employees, and high-level descriptions of ser- 
vices provided. 


Function Abstraction 


The configuration specification used by Bcfg2 
splits information into three layers. The first, called the 
metadata layer, describes function information in terms 
of clients and classes. For example, the metadata may 
describe a class of clients that include web server func- 
tionality. The second layer, called the repository, 
ascribes meaning to those descriptions. In the same 
example, the repository would contain information 
describing what “being a web server” means. The third 
layer, implementing client reconfiguration operations, 
receives configurations from the higher two layers and 
reports on execution results. While this architecture is 
Befg2 specific, many other complex configuration 
management tools are structured in a similar way [1]. 


This layered specification provides an abstrac- 
tion barrier isolating function assignment from func- 
tion description. For example, after “being a web 
server” is defined, one can easily and reliably add 
new web server instances. Once this operation is pos- 
sible, programs that generate these changes become 
possible. The addition of logic into this function deter- 
mination process introduces dramatic flexibility into 
the network. It also allows common function shifting 
operations to be automated. 


Repository semantics benefit from this abstrac- 
tion barrier as well. Several simple context-specific 
formats can be used to describe implementation 
behavior. Similarly, scripts can be written to autogen- 
erate these files, if desired. Important functionality, 
ranging from automated patch integration to service 
reconfiguration, can be implemented. 


The client-side tool receives a literal set of con- 
figuration directives from the Bcfg2 server. It com- 
pares the current operational state with the desired state 
and performs a set of state transitions, focusing on 
safely transitioning into the goal state. The client tool 
implements a small number of operations that have 
been well tested. Upon completion, the Bcfg2 client 
returns a set of statistics about the configuration state 
of the client and modifications performed. In conjunc- 
tion, these two features allow administrators to focus 
on configuration goals, while allowing the client tool 
to determine a reasonable set of operations to imple- 
ment these goals. In this way, the client tool isolates 
the upper-level users from some low-level complexity. 


Tool-Based Simplification 


Tools are intended to make tasks easier. Using 
Befg2, we found three major tasks that were vastly 
simplified. 


System rebuilds have become trivial. Statistics 
are used to verify that the configuration specification 
matches the running state of the machine. After this 
match has been verified, the machine can be rebuilt at 
the appropriate time. The previous process consisted 
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of a lot of manual verification; a second system would 
typically be used to verify functionality and swapped 
in after everything worked. 


The new machine build process has also been 
greatly accelerated and simplified. Several stock pro- 
files are available; the user is presented with a menu at 
the beginning of the build process. No other setup is 
required. Hence, non-root users are now able to build 
machines quite easily. 


The class-based system allows new profiles to be 
easily created by composing existing function groups. 
Hence, users can create new profiles whenever 
needed, without shoehorning functions into the same 
profile. In this fashion, the configuration specification 
becomes more concise and representative of the actual 
running configuration. 


Each of these tasks has become easier with the 
addition of a configuration management system. While 
these tasks could previously be performed, more exper- 
tise and privileges were frequently required. The net 
impact of these changes is to empower users to be able 
to perform tasks that were not previously possible. 


Overall Efficiency Gains 


All of these factors contributed to an impressive 
increase in efficiency. We estimate that before conver- 
sion three FTEs of time were spent on the mainte- 
nance of our workstation and server environment. 
These administrators performed a variety of tasks, 
ranging from security updates to new software instal- 
lation, and user-requested reconfigurations. After con- 
version, between one-third and one-half of an FTE is 
consumed by these activities. 


The time freed by these improvements is now 
available for a variety of activities. Large-scale infra- 
structure improvement projects are under way. More 
time for interactions with users has resulted in more 
satisfying services for users and more accurate assess- 
ments of user needs. 


Basic administration tasks remain split across 
several administrators. The use of configuration man- 
agement has also reduced the cost of task distribution. 
The existence of a central location for configuration 
specification imposes a set of expectations that allow 
administrators to find one another’s work and syn- 
chronize when needed. Most simply, all administrators 
are kept on the same page. 


Conclusions and Future Work 


Our main goal in this paper is to document the 
process and results of deploying a configuration man- 
agement tool. While the process is difficult, the out- 
comes are worthwhile. Indeed, the outcomes we expe- 
rienced have more than justified the effort involved. 
While some of the difficulties faced were certainly 
peculiar to our group, we feel that the ones docu- 
mented here are indicative of the fundamental issues 
in a change of this magnitude. 
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While our discussion may suggest that this task is 
too difficult for many groups to consider, we strongly 
believe that configuration management is an important 
technology to deploy in nearly any environment. We 
hope that our discussion of difficulties will not dissuade 
other groups but, rather, will be used to navigate an 
admittedly difficult process. 


Many administrative tasks have been vastly sim- 
plified, and much useful data can be mined from the 
configuration specification and statistics. The avail- 
ability of this data has enabled a higher level of report- 
ing and comprehension of our environment than was 
previously attainable. 


While many improvements have already been real- 
ized, further substantial efficiency gains can be achieved. 
Certainly, Bcefg2 could be improved. Several technical 
improvements relating to information representation 
require attention. Also, an interface to force client recon- 
figurations would be useful. The user interface will also 
continue to improve with more experience. 


As a group, we expect administrative model 
changes to continue, although with a less disruptive 
effect. Many processes remain that could be auto- 
mated. Also, a service that allows reconfiguration del- 
egation would be useful, allowing users to update 
aspects of configuration as appropriate. Moreover, we 
hope that administration streamlining will continue. 
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ABSTRACT 


Patching, upgrading, and maintaining operating system software is a growing management 
complexity problem that can result in unacceptable system downtime. We introduce AutoPod, a 
system that enables unscheduled operating system updates while preserving application service 
availability. AutoPod provides a group of processes and associated users with an isolated machine- 
independent virtualized environment that is decoupled from the underlying operating system 
instance. This virtualized environment is integrated with a novel checkpoint-restart mechanism 
which allows processes to be suspended, resumed, and migrated across operating system kernel 
versions with different security and maintenance patches. 


AutoPod incorporates a system status service to determine when operating system patches 
need to be applied to the current host, then automatically migrates application services to another 
host to preserve their availability while the current host is updated and rebooted. We have 
implemented AutoPod on Linux without requiring any application or operating system kernel 
changes. Our measurements on real world desktop and server applications demonstrate that 
AutoPod imposes little overhead and provides sub-second suspend and resume times that can be 
an order of magnitude faster than starting applications after a system reboot. AutoPod enables 
systems to autonomically stay updated with relevant maintenance and security patches, while 


ensuring no loss of data and minimizing service disruption. 


Introduction 


As computers become more ubiquitous in large 
corporate, government, and academic organizations, 
the total cost of owning and maintaining them is 
becoming unmanageable. Computers are increasingly 
networked, which only complicates the management 
problem, given the myriad of viruses and other attacks 
commonplace in today’s networks. Security problems 
can wreak havoc on an organization’s computing in- 
frastructure. To prevent this, software vendors fre- 
quently release patches that can be applied to address 
security and maintenance issues that have been dis- 
covered. This creates a management nightmare for 
administrators who take care of large sets of machines. 
For these patches to be effective, they need to be 
applied to the machines. It is not uncommon for sys- 
tems to continue running unpatched software long 
after a security exploit has become well-known [22]. 
This is especially true of the growing number of server 
appliances intended for very low-maintenance opera- 
tion by less skilled users. Furthermore, by reverse 
engineering security patches, exploits are being 
released as soon as a month after the fix is released, 
whereas just a couple of years ago, such exploits took 
closer to a year to create [12]. 


Even when software updates are applied to 
address security and maintenance issues, they com- 
monly result in system services being unavailable. 
Patching an operating system can result in the entire 
system having to be down for some period of time. If 


a system administrator chooses to fix an operating sys- 
tem security problem immediately, he risks upsetting 
his users because of loss of data. Therefore, a system 
administrator must schedule downtime in advance and 
in cooperation with users, leaving the computer vul- 
nerable until repaired. If the operating system is 
patched successfully, the system downtime may be 
limited to just a few minutes during the reboot. Even 
then, users are forced to incur additional inconve- 
nience and delays in starting applications again and 
attempting to restore their sessions to the state they 
were in before being shutdown. If the patch is not suc- 
cessful, downtime can extend for many hours while 
the problem is diagnosed and a solution is found. 
Downtime due to security and maintenance problems 
is not only inconvenient but costly as well. 


We present AutoPod, a system that provides an 
easy-to-use autonomic infrastructure [11] for operat- 
ing system self-maintenance. AutoPod uniquely 
enables unscheduled operating system updates of 
commodity operating systems while preserving appli- 
cation service availability during system maintenance. 
AutoPod provides its functionality without modifying, 
recompiling, or relinking applications or operating 
system kernels. This is accomplished by combining 
three key mechanisms: a lightweight virtual machine 
isolation abstraction that can be used at the granularity 
of individual applications, a checkpoint-restart mecha- 
nism that operates across operating system versions 
with different security and maintenance patches, and 
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an autonomic system status service that monitors the 
system for system faults as well as security updates. 


AutoPod provides a lightweight virtual machine 
abstraction called a POD (PrOcess Domain) that 
encapsulates a group of processes and associated users 
in an isolated machine-independent virtualized envi- 
ronment that is decoupled from the underlying operat- 
ing system instance. A pod mirrors the underlying 
operating system environment but isolates processes 
from the system by using host-independent virtual 
identifiers for operating system resources. Pod isola- 
tion not only protects the underlying system from 
compromised applications, but is crucial for enabling 
applications to migrate across operating system 
instances. Unlike hardware virtualization approaches 
that require running multiple operating system 
instances [4, 29, 30], pods provide virtual application 
execution environments within a single operating sys- 
tem instance. By operating within a single operating 
system instance, pods can support finer granularity 
isolation and can be administered using standard oper- 
ating system utilities without sacrificing system man- 
ageability. Furthermore, since it does not run an oper- 
ating system instance, a pod prevents potentially mali- 
cious code from making use of an entire set of operat- 
ing system resources. 


AutoPod combines its pod virtualization with a 
novel checkpoint-restart mechanism that uniquely 
decouples processes from dependencies on the under- 
lying system and maintains process state semantics to 
enable processes to be migrated across different 
machines. The checkpoint-restart mechanism intro- 
duces a platform-independent intermediate format for 
saving the state associated with processes and Auto- 
Pod virtualization. AutoPod combines this format with 
the use of higher-level functions for saving and restor- 
ing process state to provide a high degree of portabil- 
ity for process migration across different operating 
system versions that was not possible with previous 
approaches. In particular, the checkpoint-restart mech- 
anism relies on the same kind of operating system 
semantics that ensure that applications can function 
correctly across operating system versions with differ- 
ent security and maintenance patches. 


AutoPod combines the pod virtual machine with 
an autonomous system status service. The service 
monitors the system for system faults as well as secu- 
rity updates. When the service detects new security 
updates, it is able to download and install them auto- 
matically. If the update requires a reboot, the service 
uses the pod’s checkpoint-restart capability to save the 
pod’s state, reboot the machine into the newly fixed 
environment, and restart the processes within the pod 
without causing any data loss. This provides fast 
recovery from system downtime even when other 
machines are not available to run application services. 
Alternatively, if another machine is available, the pod 
can be migrated to the new machine while the original 
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machine is maintained and rebooted, further minimiz- 
ing application service downtime. This enables secu- 
rity patches to be applied to operating systems in a 
timely manner with minimal impact on the availability 
of application services. Once the original machine has 
been updated, applications can be returned and can 
continue to execute even though the underlying oper- 
ating system has changed. Similarly, if the service 
detects an imminent system fault, AutoPod can check- 
point the processes, migrate, and restart them on a new 
machine before the fault can cause the processes’ exe- 
cution to fail. 


We have implemented AutoPod in a prototype 
system as a loadable Linux kernel module. We have 
used this prototype to securely isolate and migrate a 
wide range of unmodified legacy and network applica- 
tions. We measure the performance and demonstrate 
the utility of AutoPod across multiple systems running 
different Linux 2.4 kernel versions using three real- 
world application scenarios, including a full KDE desk- 
top environment with a suite of desktop applications, 
an Apache/MySQL web server and database server 
environment, and a Exim/Procmail e-mail processing 
environment. Our performance results show that Auto- 
Pod can provide secure isolation and migration func- 
tionality on real world applications with low overhead. 


This paper describes how AutoPod can enable 
operating system self-maintenance by suspending, 
resuming, and migrating applications across operating 
system kernel changes to facilitate kernel maintenance 
and security updates with minimal application down- 
time. Subsequent sections describe the AutoPod virtu- 
alization abstractions, present the virtualization archi- 
tecture to support the AutoPod model, discuss the 
AutoPod checkpoint-restart mechanisms used to facili- 
tate migration across operating system kernels that 
may differ in maintenance and security updates, pro- 
vide a brief overview of the AutoPod system status 
service, provide a security analysis of the AutoPod 
system as well as examples of how to use AutoPod, 
and present experimental results evaluating the over- 
head associated with AutoPod virtualization and quan- 
tifying the performance benefits of AutoPod migration 
versus a traditional maintenance approach for several 
application scenarios. We discuss related work before 
some concluding remarks. 


AutoPod Model 


The AutoPod model is based on a_ virtual 
machine abstraction called a pod. Pods were previ- 
ously introduced in Zap [16] to support migration 
assuming the same operating system version is used 
for all systems. AutoPod extends this work to enable 
pods to provide a complete secure virtual machine 
abstraction in addition to heterogeneous migration 
functionality. A pod looks just like a regular machine 
and provides the same application interface as the 
underlying operating system. Pods can be used to run 
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any application, privileged or otherwise, without mod- 
ifying, recompiling, or relinking applications. This is 
essential for both ease-of-use and protection of the 
underlying system, since applications not executing in 
a pod offer an opportunity to attack the system. Pro- 
cesses within a pod can make use of all available oper- 
ating system services, just like processes executing in 
a traditional operating system environment. Unlike a 
traditional operating system, the pod abstraction pro- 
vides a self-contained unit that can be isolated from the 
system, checkpointed to secondary storage, migrated to 
another machine, and transparently restarted. 
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Figure 1: The AutoPod model. 





AutoPod enables server consolidation by allowing 
multiple pods to be in use on a single machine, while 
enabling automatic machine status monitoring as shown 
in Figure 1. Since each pod provides a complete secure 
virtual machine abstraction, they are able to run any 
server application that would run on a regular machine. 
By consolidating multiple machines into distinct pods 
running on a single server, one improves manageability 
by limiting the number of physical hardware and the 
number of operating system instances an administrator 
has to manage. Similarly, when kernel security holes are 
discovered, server consolidation improves manageabil- 
ity by minimizing the amount of machines that need to 
be upgraded and rebooted. The AutoPod system monitor 
further improves manageability by constantly monitor- 
ing the host system for stability and security problems. 


Since a pod does not run an operating system 
instance, it provides a virtualized machine environ- 
ment by providing a host-independent virtualized view 
of the underlying host operating system. This is done 
by providing each pod with its own virtual private 
namespace. All operating system resources are only 
accessible to processes within a pod through the pod’s 
virtual private namespace. 

A pod namespace is private in that only pro- 
cesses within the pod can see the namespace. It is pri- 
vate in that it masks out resources that are not con- 
tained within the pod. Processes inside a pod appear to 
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one another as normal processes that can communicate 
using traditional Inter-Process Communication (IPC) 
mechanisms. Other processes outside a pod do not 
appear in the namespace and are therefore not able to 
interact with processes inside a pod using IPC mecha- 
nisms such as shared memory or signals. Instead, pro- 
cesses outside the pod can only interact with processes 
inside the pod using network communication and 
shared files that are normally used to support process 
communication across machines. 


A pod namespace is virtual in that all operating 
system resources including processes, user informa- 
tion, files, and devices are accessed through virtual 
identifiers within a pod. These virtual identifiers are 
distinct from host-dependent resource identifiers used 
by the operating system. The pod virtual namespace 
provides a host-independent view of the system by 
using virtual identifiers that remain consistent 
throughout the life of a process in the pod, regardless 
of whether the pod moves from one system to another. 
Since the pod namespace is distinct from the host’s 
operating system namespace, the pod namespace can 
preserve this naming consistency for its processes 
even if the underlying operating system namespace 
changes, as may be the case in migrating processes 
from one machine to another. This consistency is 
essential to support process migration [16]. 


The pod private, virtual namespace enables 
secure isolation of applications by providing complete 
mediation to operating system resources. Pods can 
restrict what operating system resources are accessible 
within a pod by simply not providing identifiers to 
such resources within its namespace. A pod only 
needs to provide access to resources that are needed 
for running those processes within the pod. It does not 
need to provide access to all resources to support a 
complete operating system environment. An admin- 
istrator can configure a pod in the same way one con- 
figures and installs applications on a regular machine. 
Pods enforce secure isolation to prevent exploited 
pods from being used to attack the underlying host or 
other pods on the system. Similarly, the secure isola- 
tion allows one to run multiple pods from different 
organizations, with different sets of users and adminis- 
trators on a single host, while retaining the semantic of 
multiple distinct and individually managed machines. 


For example, to provide a web server, one can 
easily setup a web server pod to only contain the files 
the web server needs to run and the content it wants to 
serve. The web server pod could have its own IP 
address, decoupling its network presence from the 
underlying system. The pod can have its network 
access limited to client-initiated connections using 
firewall software to restrict connections to the pod’s IP 
address to only the ports served by the application run- 
ning within this pod. If the web server application is 
compromised, the pod limits the ability of an attacker 
to further harm the system since the only resources he 
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has access to are the ones explicitly needed by the ser- 
vice. The attacker cannot use the pod to directly initi- 
ate connections to other systems to attack them since 
the pod is limited to client-initiated connections. Fur- 
thermore, there is no need to carefully disable other 
network services commonly enabled by the operating 
system to protect against the compromised pod since 
those services, and the core operating system itself, 
reside outside of the pod’s context. 


AutoPod Virtualization 


To support the AutoPod abstraction design of 
secure and isolated namespaces on commodity operat- 
ing systems, we employ a virtualization architecture 
that operates between applications and the operating 
system, without requiring any changes to applications 
or the operating system kemel. This virtualization 
layer is used to translate between the AutoPod name- 
spaces and the underlying host operating system name- 
space. It protects the host operating system from dan- 
gerous privileged operations that might be performed 
by processes within the AutoPod, as well as protecting 
those processes from processes outside of the Auto- 
Pod. 


Pods are supported using virtualization mecha- 
nisms that translate between the pod’s resource identi- 
fiers and the operating system’s resource identifiers. 
Every resource that a process in a pod accesses is 
through a virtual private name which corresponds to 
an operating system resource identified by a physical 
name. When an operating system resource is created 
for a process in a pod, such as with process or IPC key 
creation, instead of returning the corresponding physi- 
cal name to the process, the pod virtualization layer 
catches the physical name value, and returns a virtual 
private name to the process. Similarly, any time a 
process passes a virtual private name to the operating 
system, the virtualization layer catches it and replaces 
it with the appropriate physical name. 


The key pod virtualization mechanisms used are 
a system call interposition mechanism and the chroot 
utility with file system stacking to provide each pod 
with its own file system namespace that can be sepa- 
rate from the regular host file system. Pod virtualiza- 
tion support for migration is based on Zap [16]. We 
focus here on pod virtualization support for secure vir- 
tual machine isolation. 


Because current commodity operating systems 
are not built to support multiple namespaces, AutoPod 
must take care of the security issues this causes. While 
chroot can give a set of processes a virtualized file sys- 
tem namespace, there are many ways to break out of 
the standard chrooted environment, especially if one 
allows the chroot system call to be used by processes in 
a pod. Pod file system virtualization enforces the 
chrooted environment and ensures that the pod’s file 
system is only accessible to processes within the given 
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pod by using a simple form of file system stacking to 
implement a barrier. File systems provide a permission 
function that determines if a process can access a file. 


For example, if a process tries to access a file a 
few directories below the current directory, the per- 
mission function is called on each directory as well as 
the file itself in order. If any of the calls determine that 
the process does not have permission on a directory, 
the chain of calls end. Even if the permission function 
would determine that the process would have access to 
the file itself, it must have permission to traverse the 
directory hierarchy to the file to access it. 


We implement a barrier by simply stacking a 
small pod-aware file system on top of the staging 
directory that overloads the underlying permission 
function to prevent processes running within the pod 
from accessing the parent directory of the staging 
directory, and to prevent processes running only on the 
host from accessing the staging directory. This effec- 
tively confines a process in a pod to the pod’s file sys- 
tem by preventing it from ever walking past the pod’s 
file system root. 


While any network file system can be used with 
pods to support migration, we focus on NFS because it 
is the most commonly used network file system. Pods 
can take advantage of the user identifier (UID) secu- 
rity model in NFS to support multiple security 
domains on the same system running on the same 
operating system kernel. For example, since each pod 
can have its own private file system, each pod can 
have its own /etc/passwd file that determines its list of 
users and their corresponding UIDs. In NFS, the UID 
of a process determines what permissions it has in 
accessing a file. 


By default, pod virtualization keeps process UIDs 
consistent across migration and keeps process UIDs 
the same in the pod and operating system namespaces. 
However, since the pod file system is separate from the 
host file system, a process running in the pod is effec- 
tively running in a separate security domain from 
another process with the same UID that is running 
directly on the host system. Although both processes 
have the same UID, each process is only allowed to 
access files in its own file system namespace. Simi- 
larly, multiple pods can have processes running on the 
same system with the same UID, but each pod effec- 
tively provides a separate security domain since the 
pod file systems are separate from one another. 


The pod UID model supports an easy-to-use 
migration model when a user may be using a pod on a 
host in one administrative domain and then moves the 
pod to another. Even if the user has computer accounts 
in both administrative domains, it is unlikely that the 
user will have the same UID in both domains if they 
are administratively separate. Nevertheless, pods can 
enable the user to run the same pod with access to the 
same files in both domains. 
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Suppose the user has UID 100 on a machine in 
administrative domain A and starts a pod connecting 
to a file server residing in domain A. Suppose that all 
pod processes are then running with UID 100. When 
the user moves to a machine in administrative domain 
B where he has UID 200, he can migrate his pod to the 
new machine and continue running processes in the 
pod. Those processes can continue to run as UID 100 
and continue to access the same set of files on the pod 
file server, even though the user’s real UID has 
changed. This works, even if there’s a regular user on 
the new machine with a UID of 100. While this exam- 
ple considers the case of having a pod with all pro- 
cesses running with the same UID, it is easy to see 
that the pod model supports pods that may have run- 
ning processes with many different UIDs. 


Because the root UID 0 is privileged and treated 
specially by the operating system kernel, pod virtual- 
ization treats UID 0 processes inside of a pod specially 
as well. AutoPod is required to do this to prevent pro- 
cesses running with privilege from breaking the pod 
abstraction, accessing resources outside of the pod, and 
causing harm to the host system. While a pod can be 
configured for administrative reasons to allow full priv- 
ileged access to the underlying system, we focus on the 
case of pods for running application services which do 
not need to be used in this manner. Pods do not disal- 
low UID 0 processes, which would limit the range of 
application services that could be run inside pods. 
Instead, pods provide restrictions on such processes to 
ensure that they function correctly inside of pods. 


While a process is running in user space, its UID 
does not have any affect on process execution. Its UID 
only matters when it tries to access the underlying ker- 
nel via one of the kernel entry points, namely devices 
and system calls. Since a pod already provides a virtual 
file system that includes a virtual /dev with a limited set 
of secure devices, the device entry point is already 
secured. The only system calls of concern are those that 
could allow a root process to break the pod abstraction. 
Only a small number of system calls can be used for 
this purpose. These system calls are listed and 
described in further detail in the Appendix. Pod virtual- 
ization classifies these system calls into three classes. 


The first class of system calls are those that only 
affect the host system and serve no purpose within a 
pod. Examples of these system calls include those that 
load and unload kernel modules or that reboot the host 
system. Since these system calls only affect the host, 
they would break the pod security abstraction by 
allowing processes within it to make system adminis- 
trative changes to the host. System calls that are part 
of this class are therefore made inaccessible by default 
to processes running within a pod. 

The second class of system calls are those that 
are forced to run unprivileged. Just like NFS, by 
default, squashes root on a client machine to act as 
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user nobody, pod virtualization forces privileged pro- 
cesses to act as the nobody user when they want to 
make use of some system calls. Examples of these 
system calls include those that set resource limits and 
ioctl system calls. Since system calls such as setrlimit 
and nice can allow a privileged process to increase its 
resource limits beyond predefined limits imposed on 
pod processes, privileged processes are by default 
treated as unprivileged when executing these system 
calls within a pod. Similarly, the ioctl system call is a 
system call multiplexer that allows any driver on the 
host to effectively install its own set of system calls. 
Since the ability to audit the large set of possible sys- 
tem calls is impossible given that pods may be 
deployed on a wide range of machine configurations 
that are not controlled by the AutoPod system, pod 
virtualization conservatively treats access to this sys- 
tem call as unprivileged by default. 


The final class of system calls are calls that are 
required for regular applications to run, but have 
options that will give the processes access to underly- 
ing host resources, breaking the pod abstraction. Since 
these system calls are required by applications, the 
pod checks all their options to ensure that they are 
limited to resources that the pod has access to, making 
sure they are not used in a manner that breaks the pod 
abstraction. For example, the mknod system call can be 
used by privileged processes to make named pipes or 
files in certain application services. It is therefore 
desirable to make it available for use within a pod. 
However, it can also be used to create device nodes 
that provide access to the underlying host resources. 
To limit how the system call is used, the pod system 
call interposition mechanism checks the options of the 
system call and only allows it to continue if it is not 
trying to create a device. 


Migration Across Different Kernels 


To maintain application service availability with- 
out losing important computational state as a result of 
system downtime due to operating system upgrades, 
AutoPod provide a checkpoint-restart mechanism that 
allows pods to be migrated across machines running 
different operating system kernels. Upon completion of 
the upgrade process, the respective AutoPod and its 
applications are restored on the original machine. We 
assume here that any kernel security holes on the 
unpatched system have not yet been exploited on the 
system; migrating across kernels that have already 
been compromised is beyond the scope of this paper. 
We also limit our focus to migrating between machines 
with a common CPU architecture with kernel differ- 
ences that are limited to maintenance and security 
patches. These patches often correspond to changes in 
the minor version number of the kernel. For example, 
the Linux 2.4 kernel has nearly thirty minor versions. 
Even within minor version changes, there can be sig- 
nificant changes in kernel code. Table 1 shows the 
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number of files that have been changed in various sub- 
systems of the Linux 2.4 kernel across different minor 
versions. For example, all of the files for the VM sub- 
system were changed since extensive modifications 
were made to implement a completely new page 
replacement mechanism in Linux. 


Many of the Linux kernel patches contain secu- 
rity vulnerability fixes, which are typically not sepa- 
rated out from other maintenance patches. We simi- 
larly limit our focus to where the application’s execu- 
tion semantics, such as how threads are implemented 
and how dynamic linking is done, do not change. On 
the Linux kernels this is not an issue as all these 
semantics are enforced by user-space libraries. 
Whether one uses kernel or user threads, or how 
libraries are dynamically linked into a process is all 
determined by the respective libraries on the file sys- 
tem. Since the pod has access to the same file system 
on whatever machine it is running on, these semantics 
stay the same. 


To support migration across different kernels, 
AutoPod use a checkpoint-restart mechanism that 
employs an intermediate format to represent the state 
that needs to be saved on checkpoint. On checkpoint, 
the intermediate format representation is saved and 
digitally signed to enable the restart process to verify 
the integrity of the image. Although the internal state 
that the kernel maintains on behalf of processes can be 
different across different kernels, the high-level prop- 
erties of the process are much less likely to change. We 
capture the state of a process in terms of higher-level 
semantic information specified in the intermediate for- 
mat rather than kernel specific data in native format to 
keep the format portable across different kernels. 


For example, the state associated with a UNIX 
socket connection consists of the directory entry of the 
UNIX socket file, its superblock information, a hash 
key, and so on. It may be possible to save all of this 
state in this form and successfully restore on a differ- 
ent machine running the same kernel. But this repre- 
sentation of a UNIX socket connection state is of lim- 
ited portability across different kernels. A different 
high-level representation consisting of a four tuple, 
virtual source PID, source FD, virtual destination PID, 
destination FD is highly portable. This is because the 
semantics of a process identifier and a file descriptor 
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are typically standard across different kernels, espe- 
cially across minor version differences. 


The intermediate representation format used by 
AutoPod for migration is chosen such that it offers the 
degree of portability needed for migrating between 
different kernel minor versions. If the representation 
of state is too high-level, the checkpoint-restart mech- 
anism could become complicated and impose addi- 
tional overhead. For example, the AutoPod system 
saves the address space of a process in terms of dis- 
crete memory regions called virtual memory (VM) 
areas. As an alternative, it may be possible to save the 
contents of a process’s address space and denote the 
characteristics of various portions of it in more 
abstract terms. However, this would call for an unnec- 
essarily complicated interpretation scheme and make 
the implementation inefficient. The VM area abstrac- 
tion is standard across major Linux kernel revisions. 
AutoPod view the VM area abstraction as offering suf- 
ficient portability in part because the organization of a 
process’s address space in this manner has been stan- 
dard across all Linux kernels and has never changed. 


AutoPod further support migration across differ- 
ent kernels by leveraging higher-level native kernel 
services to transform intermediate representation of the 
checkpointed image into an internal representation 
suitable for the target kernel. Continuing with the pre- 
vious example, AutoPod restore a UNIX socket con- 
nection using high-level kernel functions as follows. 
First, two new processes are created with virtual PIDs 
as specified in the four tuple. Then, each one creates a 
UNIX socket with the specified file descriptor and one 
socket is made to connect to the other. This procedure 
effectively recreates the original UNIX socket connec- 
tion without depending on many kernel internal details. 


This use of high-level functions helps in general 
portability of using AutoPod for migration. Security 
patches and minor version kernel revisions commonly 
involve modifying the internal details of the kernel 
while high-level primitives remain unchanged. As 
such services are usually made available to kernel 
modules, the AutoPod system is able to perform cross- 
kernel migration without requiring modifications to 
the kernel code. 


The AutoPod checkpoint-restart mechanism is 
also structured in such a way to perform its operations 
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Table 1: Kernel file changes within the Linux 2.4 series for 1386. 
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when processes are in a state that checkpointing can 
avoid depending on many low-level kernel details. For 
example, semaphores typically have two kinds of state 
associated with each of them: the value of the sema- 
phore and the wait queue of processes waiting to 
acquire the corresponding semaphore lock. In general, 
both of these pieces of information have to be saved 
and restored to accurately reconstruct the semaphore 
state. Semaphore values can be easily obtained and 
restored through GETALL and SETALL parameters of 
the semctl system call. But saving and restoring the 
wait queues involves manipulating kernel internals 
directly. The AutoPod mechanism avoids having to 
save the wait queue information by requiring that all 
the processes be stopped before taking the checkpoint. 
When a process waiting on a semaphore receives a stop 
signal, the kernel immediately releases the process 
from the wait queue and returns EINTR. This ensures 
that the semaphore wait queues are always empty at the 
time of checkpoint so that they do not have to be saved. 


While AutoPod can abstract and manipulate 
most process state in higher-level terms using higher- 
level kernel services, there are some parts that not 
amenable to a portable intermediate representation. 
For instance, specific TCP connection states like time- 
stamp values and sequence numbers, which do not 
have a high-level semantic value, have to be saved and 
restored to maintain a TCP connection. As this inter- 
nal representation can change, its state needs to be 
tracked across kernel versions and security patches. 
Fortunately, there is usually an easy way to interpret 
such changes across different kernels because net- 
working standards such as TCP do not change often. 
Across all of the Linux 2.4 kernels, there was only one 
change in TCP state that required even a small modifi- 
cation in the AutoPod migration mechanism. Specifi- 
cally, in the Linux 2.4.14 kernel, an extra field was 
added to TCP connection state to address a flaw in the 
existing syncookie mechanism. If configured into the 
kernel, syncookies protect an Internet server against a 
synflood attack. When migrating from an earlier ker- 
nel to a Linux-2.4.14 or later version kernel, the Auto- 
Pod system initializes the extra field in such a way 
that the integrity of the connection is maintained. In 
fact, this was the only instance across all of the Linux 
2.4 kernel versions where an intermediate representa- 
tion was not possible and the internal state had 
changed and had to be accounted for. 


To provide proper support for AutoPod virtual- 
ization when migrating across different kernels, we 
must ensure that that any changes in the system call 
interfaces are properly accounted for. As AutoPod has 
a virtualization layer using system call interposition 
mechanism for maintaining namespace consistency, a 
change in the semantics for any system call inter- 
cepted by AutoPod could be an issue in migrating 
across different kernel versions. But such changes usu- 
ally do not occur as it would require that the libraries 
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be rewritten. In other words, AutoPod virtualization is 
protected from such changes in a similar way as 
legacy applications are protected. However, new sys- 
tem calls could be added from time to time. For 
instance, across all Linux 2.4 kernels to date, there 
were two new system calls, gettid and tkill for querying 
the thread identifier and for sending a signal to a par- 
ticularly thread in a thread group, respectively, which 
needed to be accounted for to properly virtualize 
AutoPod across kernel versions. As these system calls 
take identifier arguments, they were simply inter- 
cepted and virtualized. 


Autonomic System Status Service 


AutoPod provides a generic autonomic frame- 
work for managing system state. The framework is 
able to monitor multiple sources for information and 
can use this information to make autonomic decisions 
about when to checkpoint pods, migrate them to other 
machines, and restart them. While there are many 
items that can be monitored, our service monitors two 
items in particular. First, it monitors the vendor’s soft- 
ware security update repository to ensure that the sys- 
tem stays up to date with the latest security patches. 
Second, it monitors the underlying hardware of the 
system to ensure that an imminent fault is detected 
before the fault occurs and corrupts application state. 
By monitoring these two sets of information, the auto- 
nomic system status service is able to reboot or shut- 
down the computer, while checkpointing or migrating 
the processes. This helps ensure that data is not lost or 
corrupted due to a forced reboot or a hardware fault 
propagating into the running processes. 


Many operating system vendors provide their 
users with the ability to automatically check for sys- 
tem updates and to download and install them when 
they become available. Example of these include 
Microsoft’s Windows Update service, as well as 
Debian based distribution’s security repositories. 
Users are guaranteed that the updates one gets through 
these services are genuine because they are verified 
through cryptographic signed hashes that verify the 
contents as coming from the vendors. The problem 
with these updates is that some of them require 
machine reboots; In the case of Debian GNU/Linux 
this is limited to kernel upgrades. We provide a simple 
service that monitors these security update reposito- 
ries. The autonomic service simply downloads all 
security updates, and by using the pod’s check- 
point/restart mechanism enables the security updates 
that need reboots to take effect without disrupting run- 
ning applications and causing them to lose state. 


Commodity systems also provide information 
about the current state of the system that can indicate if 
the system has an imminent failure on its hands. Sub- 
systems, such as a hard disk’s Self-Monitoring Analysis 
Reporting Technology (SMART), let an autonomic ser- 
vice monitor the system’s hardware state. SMART 


19th Large Installation System Administration Conference (LISA ’05) 53 


Reducing Downtime Due to System Maintenance and Upgrades 


provides diagnostic information, such as temperature 
and read/write error rates, on the hard drives in the sys- 
tem that can indicate if the hard disk is nearing failure. 
Many commodity computer motherboards also have the 
ability to measure CPU and case temperature, as well as 
the speeds of the fans that regulate those temperatures. 
If temperature in the machine rises too high, hardware 
in the machine can fail catastrophically. Similarly, if the 
fans fail and stop spinning, the temperature will likely 
rise out of control. Our autonomic service monitors 
these sensors and if it detects an imminent failure, will 
attempt to migrate an AutoPod to a cooler system, as 
well as shutdown the machine to prevent the hardware 
from being destroyed. 


Many administrators use an uninterruptible power 
supply to avoid having a computer lose or corrupt data 
in the event of a power loss. While one can shutdown a 
computer when the battery backup runs low, most appli- 
cations are not written to save their data in the presence 
of a forced shutdown. AutoPod, on the other hand, 
monitors UPS status and if the battery backup becomes 
low can quickly checkpoint the pod’s state to avoid any 
data loss when the computer is forced to shutdown. 


Similarly, the operating system kernel on the 
machine monitors the state of the system, and if irreg- 
ular conditions occur, such as DMA timeout or need- 
ing to reset the IDE bus, will log this occurrence. Our 
autonomic service monitors the kernel logs to discover 
these irregular conditions. When the hardware moni- 
toring systems or the kernel logs provide information 
about possible pending system failures, the autonomic 
service checkpoints the pods running on the system, 
and migrates them to a new system to be restarted on. 
This ensures state is not lost, while informing system 
administrators the a machine needs maintenance. 


Many policies can be implemented to determine 
which system a pod should be migrated to while a 
machine needs maintenance. Our autonomic service 
uses a simple policy of allowing a pod to be migrated 
around a specified set of clustered machines. The 
autonomic service gets reports at regular intervals 
from the other machines’ autonomic services that 
reports each machine’s load. If the autonomic service 
decides that it must migrate a pod, it chooses the 
machine in its cluster that has the lightest load. 


Security Analysis 


Saltzer and Schroeder [24] describe several prin- 
ciples for designing and building secure systems. 
These include: 

¢ Economy of mechanism: Simpler and smaller 
systems are easier to understand and ensure that 
they do not allow unwanted access. 

¢ Complete mediation: Systems should check 
every access to protected objects. 

¢ Least privilege: A process should only have 
access to the privileges and resources it needs 
to do its job. 
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¢ Psychological acceptability: If users are not 
willing to accept the requirements that the secu- 
rity system imposes, such as very complex 
passwords that the users are forced to write 
down, security is impaired. Similarly, if using 
the system is too complicated, users will mis- 
configure it and end up leaving it wide open. 

¢ Work factor: Security designs should force an 
attacker to have to do extra work to break the 
system. The classic quantifiable example is when 
one adds a single bit to an encryption key, one 
doubles the key space an attacker has to search. 


AutoPod is designed to satisfy these five princi- 
ples. AutoPod provides economy of mechanism using 
a thin virtualization layer based on system call inter- 
position and file system stacking that only adds a 
modest amount of code to a running system. Further- 
more, AutoPod changes neither applications nor the 
underlying operating system kernel. The modest 
amount of code to implement AutoPod makes the sys- 
tem easier to understand. Since the AutoPod security 
model only provides resources that are physically 
within the environment, it is relatively easy to under- 
stand the security properties of resource access pro- 
vided by the model. 


AutoPod provides for complete mediation of all 
resources available on the host machine by ensuring 
that all resources accesses occur through the pod’s vir- 
tual namespace. Unless a file, process, or other operat- 
ing system resource was explicitly placed in the pod 
by the administrator or created within the pod, Auto- 
Pod’s virtualization will not allow a process within a 
pod to access the resource. 


AutoPod provides a least privilege environment 
by enabling an administrator to only include the data 
necessary for each service. AutoPod can provide sepa- 
rate pods for individual services so that separate ser- 
vices are isolated and restricted to the appropriate set 
of resources. Even if a service is exploited, AutoPod 
will limit the attacker to the resources the administra- 
tor provided for that service. While one can achieve 
similar isolation by running each individual service on 
a separate machine, this leads to inefficient use of 
resources. AutoPod maintains the same least privilege 
semantic of running individual services on separate 
machines, while making efficient use of machine 
resources at hand. For instance, an administrator could 
run MySQL and Exim mail transfer services on a sin- 
gle machine, but within different pods. If the Exim pod 
gets exploited, the pod model ensures that the MySQL 
pod and its data will remain isolated from the attacker. 

AutoPod provides psychological acceptability by 
leveraging the knowledge and skills system adminis- 
trators already use to setup system environments. 
Because pods provide a virtual machine model, 
administrators can use their existing knowledge and 
skills to run their services within pods. This differs 
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from other least privilege architectures that force an 
administrator to learn new principles or complicated 
configuration languages that require a detailed under- 
standing of operating system principles. 

AutoPod increases the work factor required to 
compromise a system by not making available the 
resources that attackers depend on to harm a system 
once they have broken in. For example, services like 
mail delivery do not depend on having access to a 
shell. By not including a shell program within a mail 
delivery AutoPod, one makes it difficult for an 
attacker to get a root shell that they would use to fur- 
ther their attacks. Similarly, the fact that one can 
migrate a system away from a host that is vulnerable 
to attack increases the work an attacker would have to 
do to make services unavailable. 


AutoPod Examples 


We briefly describe two examples that help illus- 
trate how AutoPod can be used to improve application 
availability for different application scenarios. The 
application scenarios are system services, such as e- 
mail delivery and desktop computing. In both cases 
we describe the architecture of the system and show 
how it can be run within AutoPod, enabling adminis- 
trators to reduce downtime in the face of machine 
maintenance. We also discuss how a system admin- 
istrator can setup and use pods. 


System Services 


Administrators like to run many services on a 
single machine. By doing this, they are able to benefit 
from improved machine utilization, but at the same 
time give each service access to many resources they 
do not need to perform their job. A classic example of 
this is e-mail delivery. E-mail delivery services, such 
as Exim, are often run on the same system as other 
Internet services to improve resource utilization and 
simplify system administration through server consoli- 
dation. However, services such as Exim have been 
easily exploited by the fact that they have access to 
system resources, such as a shell program, that they do 
not need to perform their job. 


For e-mail delivery, AutoPod can isolate e-mail 
delivery to provide a significantly higher level of 
security in light of the many attacks on mail transfer 
agent vulnerabilities that have occurred. Consider iso- 
lating an Exim service, the default Debian mail trans- 
fer agent, installation. Using AutoPod, Exim can exe- 
cute in a resource restricted pod, which isolates e-mail 
delivery from other services on the system. Since pods 
allow one to migrate a service between machines, the 
e-mail delivery pod is migratable. If a fault is discov- 
ered in the underlying host machine, the e-mail deliv- 
ery service can be moved to another system while the 
original host is patched, preserving the availability of 
the e-mail service. 

With this e-mail delivery example, a simple system 
configuration can prevent the common buffer overflow 
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exploit of getting the privileged server to execute a local 
shell. This is done by just removing shells from within 
the Exim pod, thereby limiting the amateur attacker’s 
ability to exploit flaws while requiring very little addi- 
tional knowledge about how to configure the service. 
AutoPod can further automatically monitor system status 
and checkpoint the Exim pod if a fault is detected to 
ensure that no data is lost or corrupted. Similarly, in the 
event that a machine has to be rebooted, the service can 
automatically be migrated to a new machine to avoid any 
service downtime. 


A common maintenance problem system admin- 
istrators face is that forced machine downtime, for 
example due to reboots, can cause a service to be 
unavailable for a period of time. A common way to 
avoid this problem is to throw multiple machines at 
the problem. By providing the service through a clus- 
ter of machines, system administrators can upgrade 
the individual machines in a rolling manner. This 
enables system administrators to upgrade the systems 
providing the service while keeping the service avail- 
able. The problem with this solution is that system 
administrators need to throw more machines at the 
problem than they might need to provide the service 
effectively, thereby increasing management complex- 
ity as well as cost. 


AutoPod in conjunction with hardware virtual 
machine monitors improves this situation immensely. 
Using a virtual machine monitor to provide two virtual 
machines on a single host, AutoPod can then run a pod 
within a virtual machine to enable a single node main- 
tenance scenario that can decrease costs as well man- 
agement complexity. During regular operation, all 
application services run within the pod on one virtual 
machine. When one has to upgrade the operating sys- 
tem in the running virtual machine, one brings the sec- 
ond virtual machine online and migrates the pod to the 
new virtual machine. 


Once the initial virtual machine is upgraded and 
rebooted, the pod can be migrated back to it. This 
reduces costs as only a single physical machine is 
needed. This also reduces management complexity as 
only one virtual machine is in use for the majority of 
the time the service is in operation. Since AutoPod 
runs unmodified applications, any application service 
that can be installed can make use of AutoPod’s ability 
to provide general single node maintenance. 


Desktop Computing 


As personal computers have become more ubiq- 
uitous in large corporate, government, and academic 
organizations, the total cost of owning and maintain- 
ing them is becoming unmanageable. These computers 
are increasingly networked which only complicates 
the management problem. They need to be constantly 
patched and upgraded to protect them, and their data, 
from the myriad of viruses and other attacks common- 
place in today’s networks. 
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To solve this problem, many organizations have 
turned to thin-client solutions such as Microsoft’s 
Windows Terminal Services and Sun’s Sun Ray. Thin 
clients give administrators the ability to centralize 
many of their administrative duties as only a single 
computer or a cluster of computers needs to be main- 
tained in a central location, while stateless client 
devices are used to access users’ desktop computing 
environments. While thin-client solutions provide 
some benefits for lowering administrative costs, this 
comes at the loss of semantics users normally expect 
from a private desktop. For instance, users who use 
their own private desktop expect to be isolated from 
their coworkers. However, in a shared thin-client envi- 
ronment, users share the same machine. There may be 
many shared files and a user’s computing behavior can 
impact the performance of other users on the system. 


While a thin-client environment minimizes the 
machines one has to administrate, the centralized 
servers still need to be administrated, and since they 
are more highly utilized, management becomes more 
difficult. For instance, on a private system one only 
has to schedule system maintenance with a single user, 
as reboots will force the termination of all programs 
running on the system. However, in a thin-client envi- 
ronment, one has to schedule maintenance with all the 
users on the system to avoid having them lose any 
important data. 


AutoPod enables system administrators to solve 
these problems by allowing each user to run a desktop 
session within a pod. Instead of users directly sharing 
a single file system, AutoPod provides each pod with 
a composite of three file systems: a shared read-only 
file system of all the regular system files users expect 
in their desktop environments, a private writable file 
system for a user’s persistent data, and a private 
writable file system for a user’s temporary data. By 
sharing common system files, AutoPod provides cen- 
tralization benefits that simplify system administra- 
tion. By providing private writable file systems for 
each pod, AutoPod provides each user with privacy 
benefits similar to a private machine. 


Coupling AutoPod virtualization and isolation 
mechanisms with a migration mechanism can provide 
scalable computing resources for the desktop and 
improve desktop availability. If a user needs access to 
more computing resources, for instance while doing 
complex mathematical computations, AutoPod can 
migrate that user’s session to a more powerful machine. 
If maintenance needs to be done on a host machine, 
AutoPod can migrate the desktop sessions to other 
machines without scheduling downtime and without 
forcefully terminating any programs users are running. 


Setting Up and Using AutoPod 


To demonstrate how simple it is to setup a pod to 
run within the AutoPod environment, we provide a 
step by step walkthrough on how one would create a 
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new pod that can run the Exim mail transfer agent. 
Setting up AutoPod to provide the Exim pod on Linux 
is straightforward and leverages the same skill set and 
experience system administrators already have on stan- 
dard Linux systems. AutoPod is started by loading its 
kernel module into a Linux system and using its user- 
level utilities to setup and insert processes into a pod. 


Creating a pod’s file system is the same as creat- 
ing a chroot environment. Administrators that have 
experience creating a minimal environment, that just 
contains the application they want to isolate, do not 
need to do any extra work. However, many adminis- 
trators do not have experience creating such an envi- 
ronment and therefore need an easy way to create an 
environment to run their application in. These admin- 
istrators can take advantage of Debian’s debootstrap 
utility that enables a user to quickly setup an environ- 
ment that’s the equivalent of a base Debian installa- 
tion. An administrator would do a debootstrap stable 
[pod to install the most recently released Debian sys- 
tem into the /pod directory. While this will also include 
many packages that are not required by the installa- 
tion, it provides a small base to work from. An admin- 
istrator can remove packages, such as the installed 
mail transfer agent, that are not needed. 


To configure Exim, an administrator edits the 
appropriate configuration files within the /pod/etc/ 
exim4/ directory. To run Exim in a pod, an administra- 
tor does mount -o bind /pod /autopod/exim/root to loop- 
back mount the pod directory onto the staging area 
directory where AutoPod expects it. autopod add exim 
is used to create a new pod named exim which uses 
/autopod/exim/root as the root for its file system. Finally, 
autopod addproc exim /usr/sbin/exim4 is used to start 
Exim within the pod by executing the /usr/sbin/exim4 
program, which is actually located at /autopod/exim/ 
root/usr/sbin/exim4. 


AutoPod isolates the processes running within a 
pod from the rest of the system, which helps contain 
intrusions if they occur. However, since a pod does not 
have to be maintained by itself, but can be maintained 
in the context of a larger system, one can also prune 
down the environment and remove many programs that 
an attacker could use against the system. For instance, 
if an Exim pod has no need to run any shell scripts, 
there is no reason an administrator has to leave pro- 
grams such as /bin/bash, /bin/sh and /bin/dash within the 
environment. One issue is that these programs are nec- 
essary if the administrator wants to be able to simply 
upgrade the package in the future via normal Debian 
methods. Since it is simple to recreate the environment, 
one approach would be to remove all the programs that 
are not wanted within the environment and recreate the 
environment when an upgrade is needed. Another 
approach would be to move those programs outside of 
the pod, such as by creating a /pod-backup directory. To 
upgrade the pod using the normal Debian package 
upgrade methods, the programs can then be moved 
back into the pod file system. 
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If an administrator wants to manually reboot the 
system without killing the processes within this Exim 
pod, the administrator can first checkpoint the pod to 
disk by running autopod checkpoint exim -o /exim.pod, 
which tells AutoPod to checkpoint the processes asso- 
ciated with the exim pod to the file /exim.pod. The sys- 
tem can then be rebooted, potentially with an updated 
kernel. Once it comes back up, the pod can be 
restarted from the /exim.pod file by running autopod 
restart exim -i /exim.pod. These mechanisms are the same 
as those used by the AutoPod system status service for 
controlling the checkpointing and migration of pods. 


Standard Debian facilities for installing packages 
can be used for running other services within a pod. 
Once the base environment is setup, an administrator 
can chroot into this environment by running chroot /pod 
to continue setting it up. By editing the /etc/apt/ 
sources.list file appropriately and running apt-get update, 
an administrator will be able to install any Debian 
package into the pod. In the Exim example, Exim does 
not need to be installed since it is the default MTA and 
already included in the base Debian installation. If one 
wanted to install another MTA, such as Sendmail, one 
could run apt-get install sendmail, which will download 
Sendmail and all the packages needed to run it. This 
will work for any service available within Debian. An 
administrator can also use the dpkg --purge option to 
remove packages that are not required by a given pod. 
For instance, in running an Apache web server in a 
pod, one could remove the default Exim mail transfer 
agent since it is not needed by Apache. 


Experimental Results 


We implemented AutoPod as a loadable kernel 
module in Linux, that requires no changes to the 
Linux kernel, as well as a user space system status 
monitoring service. We present some experimental 
results using our Linux prototype to quantify the over- 
head of using AutoPod on various applications. Exper- 
iments were conducted on a trio of IBM Netfinity 


Name 













shmget-shmctl 













semget-semctl 
fork-exit 












Apache 





Postmark 





getpid average getpid runtime 350 ns 
average runtime for the FONREAD ioctl 427 ns 


IPC Shared memory segment holding an integer is created and | 3361 ns 
removed 


IPC Semaphore variable is created and removed 1370 ns 


process forks and waits for child which calls exit immediately 





process forks and waits for child to run /bin/sh to run a pro- 
gram that prints “hello world” then exits 


Runs Apache under load and measures average request time 1.2 ms 
Linux Kernel compile with up to 10 process active at one time 224.5 s 


Use Postmark Benchmark to simulate Exim performance .002 s 


“TPC-W like” interactions benchmark 8.33 s 
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4500R machines, each with a 933 Mhz Intel Pentium- 
Il] CPU, 512 MB RAM, 9.1 GB SCSI HD and a 100 
Mbps Ethernet connected to a 3Com Superstack II 
3900 switch. One of the machines was used as an NFS 
server from which directories were mounted to con- 
struct the virtual file system for the AutoPod on the 
other client systems. The clients ran different Linux 
distributions and kernels, one machine running Debian 
Stable with a Linux 2.4.5 kernel and the other running 
Debian Unstable with a Linux 2.4.18 kernel. 


To measure the cost of AutoPod virtualization, 
we used a range of micro benchmarks and real applica- 
tion workloads and measured their performance on our 
Linux AutoPod prototype and a vanilla Linux system. 
Table 2 shows the seven micro-benchmarks and four 
application benchmarks we used to quantify AutoPod 
virtualization overhead as well as the results for a 
vanilla Linux system. To obtain accurate measure- 
ments, we rebooted the system between measurements. 
Additionally, the system call micro-benchmarks di- 
rectly used the TSC register available on Pentium 
CPUs to record timestamps at the significant measure- 
ment events. Each timestamp’s average cost was 58 ns. 
The files for the benchmarks were stored on the NFS 
Server. All of these benchmarks were performed in a 
chrooted environment on the NFS client machine run- 
ning Debian Unstable with a Linux 2.4.18 kernel. Fig- 
ure 4 shows the results of running the benchmarks 
under both configurations, with the vanilla Linux con- 
figuration normalized to one. Since all benchmarks 
measure the time to run the benchmark, a small num- 
ber is better for all benchmarks results. 


The results in Figure 2 show that AutoPod virtu- 
alization overhead is small. AutoPod incurs less than 
10% overhead for most of the micro-benchmarks and 
less than 4% overhead for the application workloads. 
The overhead for the simple system call getpid bench- 
mark is only 7% compared to vanilla Linux, reflecting 
the fact that AutoPod virtualization for these kinds of 
system calls only requires an extra procedure call and 





Linux 


44.7 us 
3.89 ms 












Table 2: Application benchmarks. 
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a hash table lookup. The most expensive benchmarks 
for AutoPod is semgettsemct! which took 51% longer 
than vanilla Linux. The cost reflects the fact that our 
untuned AutoPod prototype needs to allocate memory 
and do a number of namespace translations. 


Plain Linux 2 


AutoPod 


Normalized performance 





Make = 


= gs & 
So «& 3 
= = & 
£ @& 


forkexec 
forkexit 
semget 
shmget 
PostMark 
Apache 
MySQL 


Figure 2: AutoPod virtualization overhead. 


The ioctl benchmark also has high overhead, 
because of the 12 separate assignments it does to pro- 
tect the call against malicious root processes. This is 
large compared to the simple FIONREAD ioctl that just 
performs a simple dereference. However, since the ioctl 
is simple, we see that it only adds 200 ns of overhead 
over any ioctl. For real applications, the most overhead 
was only four percent which was for the Apache work- 
load, where we used the http_load benchmark [18] to 
place a parallel fetch load on the server with 30 clients 
fetching at the same time. Similarly, we tested MySQL 
as part of a web-commerce scenario outlined by TPC- 
W with a bookstore servlet running on top of Tomcat 
with a MySQL back-end. The AutoPod overhead for 
this scenario was less than 2% versus vanilla Linux. 


To measure the cost of AutoPod migration and 


demonstrate the ability of AutoPod to migrate real appli- 
cations, we migrated the three application scenarios; an 







Desktop | Xvne — VNC 3.3.3r2 X Server 












to a remote host 


KDE 


es a Normal 
Name Applications Startup 


femal | Exim336 
Apache 1.3.26 and MySQL 4.0.14. 





KDE — Entire KDE 2.2.2 environment, including window manager, 
panel and assorted background daemon and utilities 


SSH — openssh 3.4p1 client inside a KDE konsole terminal connected 


Shell — The Bash 2.05a shell running in a konsole terminal 
KGhostView — A PDF viewer with a 450 KB 16 page PDF file loaded. 
Konqueror — A modern standards compliant web browser that is part of 


K Office — The KDE word processor and spreadsheet programs 
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email delivery service using Exim and Procmail, a web 
content delivery service using Apache and MySQL, and 
a KDE desktop computing environment. Table 3 
described the configurations of the application scenarios 
we migrated, as well as showing the time it takes to 
startup on a regular Linux system. To demonstrate our 
AutoPod prototype’s ability to migrate across Linux ker- 
nels with different minor versions, we checkpointed 
each application workload on the 2.4.5 kernel client 
machine and restarted it on the 2.4.18 kernel machine. 
For these experiments, the workloads were check- 


pointed to and restarted from local disk. 

Check 
Case Point Compr’d 
E-mail 284KB| 84KB 
5.3MB | 332 KB 


Table 4: AutoPod migration costs. 











Table 4 shows the time it took to checkpoint and 
restart each application workload. In addition to these, 
migration time also has to take into account network 
transfer time. As this is dependent on the transport 
medium, we include the uncompressed and compressed 
checkpoint image sizes. In all cases, checkpoint and 
restart times were significantly faster than the regular 
startup times listed in Table 5, taking less than a second 
for both operations, even when performed on separate 
machines or across a reboot. We also show that the 
actual checkpoint images that were saved were modest 
in size for complex workloads. For example, the Desk- 
top pod had over 30 different processes running, provid- 
ing the KDE desktop applications, as well as substantial 
underlying window system infrastructure, including 
inter-application sharing, a rich desktop interface man- 
aged by a window manager with a number of applica- 
tions running in a panel such as the clock. Even with all 
these applications running, they checkpoint to a very 
reasonable 35 MB uncompressed for a full desktop 






19s 


Table 3: Application scenarios. 
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environment. Additionally, if one needed to transfer the 
checkpoint images over a slow link, Table 6 shows that 
they can be compressed very well with the bzip2 com- 
pression program. 


Related Work 


Virtual machine monitors (VMMs) have been 
used to provide secure isolation [4, 29, 30], and have 
also been used to to migrate an entire operating system 
environment [25]. Unlike AutoPod, standard VMMs 
decouple processes from the underlying machine hard- 
ware, but tie them to an instance of an operating sys- 
tem. As a result, VMMs cannot migrate processes 
apart from that operating system instance and cannot 
continue running those processes if the operating sys- 
tem instance ever goes down, such as during security 
upgrades. In contrast, AutoPod decouples process exe- 
cution from the underlying operating system which 
allows it to migrate processes to another system when 
an operating system instance is upgraded. VMMs have 
been proposed to support online maintenance of sys- 
tems [14] by having a microvisor that supports at most 
two virtual machines running on the machine at the 
same time, effectively giving each physical machine 
the ability to act as its own hot spare. However, this 
proposal explicitly depends on AutoPod migration 
functionality yet does not provide it. 


A number of other approaches have explored the 
idea of virtualizing the operating system environment 
to provide application isolation. FreeBSD’s Jail mode 
[10] provides a chroot like environment that processes 
can not break out of. However, since Jail is limited in 
what it can do, such as the fact it does not allow IPC 
within a jail [9] many real world application will not 
work. More recently, Linux Vserver [1] and Solaris 
Zones [19] offer a similar virtual machine abstraction 
as AutoPod pods, but require substantial in-kernel 
modifications to support the abstraction. They do not 
provide isolation of migrating applications across 
independent machines, and have no support for main- 
taining application availability in the presence of oper- 
ating system maintenance and security upgrades. 

Many systems have been proposed to support 
process migration [2, 3, 6, 7, 8, 13, 15, 17, 20, 21, 23, 
26], but do not allow migration across independent 
machines running different operating system versions. 
TUI [27] provides support for process migration across 
machines running different operating systems and hard- 
ware architectures. Unlike AutoPod, TUI has to compile 
applications on each platform using a special compiler 
and does not work with unmodified legacy applications. 
AutoPod builds on a pod abstraction introduced in Zap 
[16] to support transparent migration across systems 
running the same kernel version. Zap does not address 
security issues or heterogeneous migration. AutoPod 
goes beyond Zap in providing a complete, secure virtual 
machine abstraction for isolating processes, finer-grain 
mechanisms for isolating application components, and 
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transparent migration across minor kernel versions, 
which is essential for providing application availability 
in the presence of operating system security upgrades. 


Replication in clustered systems can provide the 
ability to do rolling upgrades. By leveraging many 
nodes, individual nodes can be taken down for mainte- 
nance, without significantly impacting the load the 
cluster can handle. For example, web content is com- 
monly delivered by multiple web servers behind a front 
end manager. This front end manager enables an 
administrator to bring down back end web servers for 
maintenance as it will only direct requests to the active 
web servers. This simple solution is effective because it 
is easy to replicate web servers to serve the same con- 
tent. While this model works fine for web server loads, 
as the individual jobs are very short, it does not work 
for long running jobs, such as a user’s desktop. In the 
web server case, replication and upgrades are easy to do 
since only one web server is used to serve any individ- 
ual request and any web server can be used to serve any 
request. For long running stateful applications, such as 
a user’s desktop, requests cannot be arbitrarily redi- 
rected to any desktop computing environment as each 
user’s desktop session is unique. While specialized 
hardware support could be used to keep replicas syn- 
chronized, by having all of them process all operations, 
this is prohibitively expensive for most workloads and 
does not address the problem of how to resynchronize 
the replicas in the presence of rolling upgrades. 


Another possible solution to this problem is allow- 
ing the kernel to be hot plugable. While micro-kernels 
are not prevalent, they contain this ability to upgrade 
their parts on the fly. More commonly, many modern 
monolithic kernels have kernel modules that can be 
inserted and removed dynamically. This can allow one 
to upgrade parts of a monolithic kernel without requir- 
ing any reboots. The Nooks [28] system extends this 
concept by enabling kernel drivers and other kernel 
functionality, such as file systems, to be isolated into 
their own protection domain to help isolate faults in 
kernel code and provide a more reliable system. How- 
ever, in all of these cases, their is still a base kernel on 
the machine that cannot be replaced without a reboot. If 
one has to replace that part, all data would be lost. 


The K42 operating system has the ability to be 
dynamically updated [5]. This functionality enables 
software patches to be applied to a running kernel even 
in the presence of data structure changes. However, it 
requires a completely new operating system design 
and does not work with any commodity operating sys- 
tem. Even on K42, it is not yet possible to upgrade the 
kernel while running realistic application workloads. 


Conclusions 


The AutoPod system provides an operating system 
virtualization layer that decouples process execution 
from the underlying operating system, by running the 
process within a pod. Pods provide an easy-to-use 
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lightweight virtual machine abstraction that can securely 
isolate individual applications without the need to run an 
operating system instance in the pod. Furthermore, Auto- 
Pod can be transparently migrate isolated applications 
across machines running different operating system ker- 
nel versions. This enables security patches to be applied 
to operating systems in a timely manner with minimal 
impact on the availability of application services. It also 
preserves secure isolation of untrusted applications in the 
presence of migration. 


We have implemented AutoPod on Linux with- 
out requiring any application or operating system ker- 
nel changes. We demonstrated how pods can be used 
to enable autonomic machine maintenance and 
increase availability for a range of applications, includ- 
ing e-mail delivery, web servers with databases and 
desktop computing. Our measurements on real world 
applications demonstrate that AutoPod imposes little 
overhead, provides sub-second suspend and resume 
times that can be an order of magnitude faster than 
starting applications after a system reboot, and enables 
systems to autonomously stay updated with relevant 
maintenance and security patches, while ensuring no 
loss of data and minimizing service disruption. 
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Appendix 


To isolate regular Linux processes within a pod, 
AutoPod interposes on a number of system calls. 
Below we provide a complete list of the small number 
of system calls that require more than plain virtualiza- 
tion. We give the reasoning for the interposition and 
what functionality was changed from the base system 
call. Most system calls do not require more than sim- 
ple virtualization to ensure isolation because virtual- 
ization of the resources itself takes care of the isola- 
tion. For example, the kill system call can not signal a 
processes outside of a pod because the virtual private 
namespace will not map them and therefore it cannot 
reference it. 


Host Only System Calls 


1. mount — If a user within a regular pod is able to 
mount a file system, they could mount a file sys- 
tem with device nodes already present and thus 
would be able to access the underlying system 
directly in a manner that is not controlled by 
AutoPod. Therefore, regular pod processes are 
prevented from using this system call. 

2. stime, adjtimex — These system call enable a privi- 
leged process to adjust the host’s clock. If a user 
within a regular pod could call this system call 
they would cause a change on the host. Therefore 
regular pod processes are prevented from using 
this system call. 

3. acct— This system call sets what file on the host 
BSD process accounting information should be 
written to. As this is host specific functionality, 
AutoPod prevents regular pod processes from 
using this system call. 

4. swapon, swapoff — These system calls control 
swap space allocation. Since these system calls 
are host specific and have no use within a regu- 
lar pod, AutoPod prevents regular pod pro- 
cesses from calling these system calls. 

5. reboot — This system call can cause the system 
to reboot or change Ctrl-Alt-Delete functional- 
ity and therefore serves no place inside a regu- 
lar pod. AutoPod therefore disallows regular 
pod processes from calling it. 

6. ioperm, iop!| — These system calls enable a privi- 
leged process to gain direct access to underly- 
ing hardware resources. Since regular pod pro- 
cesses do not access hardware directly, Auto- 
Pod prevents regular pod process from calling 
these system calls. 
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7. create_module, init_module, delete_module, query_ 


module — These system calls are only related to 
inserting and removing kernel modules. As this 
is a host specific function, AutoPod prevents 
regular pod processes from calling these system 
calls. 


. sethostname, setdomainname — These system call 


sets the name for the underlying host. AutoPod 
wraps these system calls to save it as a pod spe- 
cific name and allows each pod to call it inde- 
pendently. 


. nfsservctl - This system call can enable a privi- 


leged process inside a pod to change the host’s 
internal NFS server. AutoPod therefore pre- 
vents a process within a regular pods from call- 
ing this system call. 


Root Squashed System Calls 


1. nice, setpriority, sched_setscheduler — These sys- 


tem calls lets a process change its priority. If a 
process is running as root (UID 0), it can 
increase its priority and freeze out other pro- 
cesses on the system. Therefore, AutoPod pre- 
vents any regular pod process from increasing 
its priority. 


. ioctl - This system call is a syscall demulti- 


plexer that enables kernel device drivers and 
subsystems to add their own functions that can 
be called from user space. However, as func- 
tionality can be exposed that enables root to 
access the underlying host, all system call 
beyond a limited audited safe set are squashed 
to user nobody, similar to what NFS does. 


. Setrlimit — this system call enables processes run- 


ning as uid 0 to raise their resource limits 
beyond what was preset, thereby enabling them 
to disrupt other processes on the system by 
using too much resources. AutoPod therefore 
prevents regular pod processes from using this 
system call to increase the resources available 
to them. 


. mlock, mlockall — These system calls enable a privi- 


leged process to pin an arbitrary amount of mem- 
ory, thereby enabling a pod process to lock all of 
available memory and starve all the other pro- 
cesses on the host. AutoPod therefore squashes a 
privileged processes to user nobody when it 
attempts to call this system call to treat it like a 
regular process. 


Option Checked System Calls 
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1. mknod — This system call enables a privileged 


user to make special files, such pipes, sockets 
and devices as well as regular files. Since a privi- 
leged process needs to make use of such func- 
tionality, the system call cannot be disabled. 
However, if the process could create a device it 
be creating an access point to the underlying host 
system. Therefore when a regular pod process 
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makes use of this system call, the options are 
checked to prevent it from creating a device spe- 
cial file, while allowing the other types through 
unimpeded. 
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ABSTRACT 


The UNIX flavors in use today have so much in common that centralized management of 
UNIX systems has become almost standard. Since Mac OS X is based on BSD-UNIX it is a 
promising candidate for integration into a centrally managed UNIX environment. 


Starting from generic administration concepts, this paper develops an integrated management 
concept that handles fully automated installation and configuration of hosts. The concept includes 
a centralized application management system for console and graphical Mac OS X applications. 


The management concept is then implemented based exclusively on standard UNIX tools. 
The necessary extensions of these tools to make Mac OS X conform to UNIX standards are 
presented, including a proxy tool to forward AppleEvents which facilitate the interprocess 
communication for centrally managed graphical Mac OS X applications. 


Introduction 


The increasing diversity of hardware and soft- 
ware makes system management more difficult. 
Shorter life-cycles of computer systems require more 
frequent upgrades or replacement of hardware and as a 
consequence, the installed computers on a large site 
rarely are uniform in hardware but rather split into sev- 
eral uniform clusters. Automated management of such 
an environment is challenging as complexity grows 
with each new configuration of hardware and software. 





is Peou : eres sare, 
Figure 1: Structure of the Mac OS X operating sys- 


tem (from [2]). 


Through integrated management of the clusters 
common tools and common configuration information 
can be reused across the clusters thus reducing the amount 
of information required to fully understand the entire site. 
With Mac OS X being a member of the UNIX family 
integrated management of Mac OS X in a UNIX environ- 
ment comes within reach. In this paper, design, 


implementation and deployment of such a system will be 
layed out built on some basic principles. The result will 
be a management system for UNIX systems that can be 
used to manage network-based as well as stand-alone sys- 
tems. 


In system management three basic principles 
keep appearing in tools and methods [1], and these 
shall be used as orientation for our integrated manage- 
ment system: 

¢ Reproducibility ensures that the same action 
produces identical results. Automation, a way 
to implement reproducibility, helps to exclude 
human error in repetitive tasks. 

© Comprehensibility of all actions is necessary for 
the administrator especially when troubleshoot- 
ing or modifying the configuration. 

e Avoidance of Redundancy helps to keep data 
consistent and thus easier to manage. 


Mac OS X is UNIX Plus... 


Mac OS X unites the strengths in UI and applica- 
tion design of previous Macintosh operating systems 
with the stability and flexibility of a modern UNIX 
platform. Integrating Mac OS X into a general UNIX 
environment requires a closer look at the operating 
system since not all parts in Mac OS X have their ori- 
gin in the UNIX world: 

¢ Darwin: Darwin is based on Free-BSD includ- 
ing the standard UNIX network clients and 
servers as well as the usual user space utilities. 

The appearance of some daemons and configu- 

ration files have been modified to match with 

the rest of Mac OS X. 

© Quartz, OpenGL, QuickTime: Instead of 
relying on the UNIX XII standard, Apple 
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decided to build an alternative graphic system. 

° Classic: The Classic environment provides 
emulation support for native pre-OSX applica- 
tions. These applications only work with 
Apple’s HFS/HFS-+ filesystem. 

¢ Carbon: The Carbon library framework pro- 
vides compatibility to pre-OSX system calls at 
source code level. Carbonized applications also 
run on non-HFS filesystems through Carbon’s 
HFS emulation although with reduced stability. 

¢ Cocoa: Providing an entire new standard for 
application development, Cocoa is based on 
modern, UNIX compatible technologies like 
XML and Java. 

¢ Aqua: The top layer of Figure | represents the 
graphical user interface on which the different 
GUI applications run. 


Looking at the entire operating system, the Dar- 
win roots as well as modern Cocoa-based applications 
are fully compatible with the rest of the UNIX world 
even though Apple did not use the X11 standards in 
their graphics system. Thus the key challenge in man- 
aging a Mac OS X system as a UNIX is the handling 
of legacy applications and their specialties. 


Review of Available Tools 


Several system management tools are available 
under Mac OS X that focus on three different manage- 
ment areas: 

¢ Installation of the operating system 
¢ Configuration of operating system and appli- 
cations 
¢ Software distribution or installation onto an 
installed host 
The management tools need to be applicable on classic 
UNIX flavors while supporting the Mac OS X specific 
extensions, e.g., legacy application support. Several 
candidate tools were considered, and their strengths and 
weaknesses will be discussed in the following sections. 


Installation Tools 


Several strategies can be chosen to install an 
operating system onto a target host. Network-based 
installation allows access to centralized services and is 
logistically efficient. Therefore only network-based 
installer methods are considered in the choice of tools. 


NetInstall 


Apple provides a native mechanism [3] to install 
multiple client machines based on the installation of 
packages (.pkg bundles [4]). The target host of an 
installation is net-booted from a modified disk image 
that will start an installer system. The installer then 
installs packages supplied on the booted image. 


The package-based approach of NetlInstall yields 
reproducible and certainly comprehensible results since 
all changes on the installed system have their source in 
one of the installed packages. As the installer image can 
only contain a single configuration several NetInstall 
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images are required when managing multiple host con- 
figurations. Consequently, each image would redun- 
dantly contain commonly installed packages making 
the management of NetInstall a difficult task in a het- 
erogeneous environment. 


Net-Restore 


Mike Bombich’s NetRestore [5] is a suite of GUI 
tools that are based on ASR, Apple’s image manage- 
ment tool [6]. An installation is started by net-booting 
a target host into the NetRestore installer in which the 
administrator selects the image to be restored to the 
local disk. The individual images are supplied through 
a network share along with post-installation scripts for 
the individual configuration of the installed host. 


Since the restored system is identical with the 
source image the installation itself clearly is repro- 
ducible. However creation and maintenance of the 
source image is done by hand and the final system can 
not be comprehensible as a whole. An administrator nei- 
ther explicitly sees why a system is in the current state 
nor completely understands the consequences of each 
manual step during assembly. As with NetInstall in the 
previous section the management of several configura- 
tions implies the use of multiple disk images introducing 
redundancy between the manually maintained images. 


Sun Solaris Jumpstart 


Network based installation is done by net-boot- 
ing an installation target into the Sun Solaris Jumpstart 
[7] system. Once booted Jumpstart uses DHCP and 
DNS to determine the correct configuration list of 
packages and appropriate pre/post-processing scripts. 


The Jumpstart configuration concept is simple 
and yet capable of comprehensibly handling individual 
configurations. Its design for completely unattended 
installation makes the Jumpstart system reproducible. 


Configuration Tools 


To reproducibly maintain the configuration of 
systems, especially in a heterogeneous environment, 
automated tools are essential. However to provide the 
necessary comprehensibility, configuration information 
consisting of a large number of modifications for a tar- 
get system must be structured into modules. By postu- 
lating module integrity, i.e., that no module destroys 
the modifications of another, reuse of modules becomes 
possible, thus controlling unwanted redundancy. The 
following three tools fulfill this basic requirement. 
Cfengine 

One of the best known tools comes from Mark 
Burgess of Oslo University College. Cfengine [8] is a 
highly flexible scripting system that deducts its configu- 
ration based on the context of a managed host. Cfengine 
supports various UNIX flavors including Mac OS X 
and is equipped with its own file sharing mechanism. 


Provided that Cfengine is run in the same con- 
text, reproducible results can be expected, and modu- 
larization is provided through the classes construct. 
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However, since Cfengine does not enforce integrity of 
actions or classes, configuration scripts can easily 
exceed the state of comprehensibility. 


Radmind 


Radmind [9] is available on various UNIX 
dialects including Mac OS X. Designed for ease of 
use, Radmind implements a capture and replay strat- 
egy resulting in a tree of dependent /oad sets consist- 
ing of files modified during a capture session. 


The capture and replay processes alone are com- 
prehensible and reproducible. However the dependen- 
cies between load sets restrict the replay of features 
between different lines of history. As these restrictions 
are not managed by Radmind the replay of an unsuit- 
able load set impairs the reproducibility of functional- 
ity and the comprehensibility of the configuration. 


Template Tree 2 


Template Tree 2(TeTre2) [10] is used for the 
administration of UNIX clusters. The configuration of 
a system is managed using simple file operations 
structured into integrity-preserving modules. A subset 
of Cfengine functionality is used to propagate the con- 
figuration to a target system. 


Although dependencies between features in a 
TeTre2 configuration exist, these dependencies are 
functional rather than historical. This allows features 
to be recombined freely without impact on compre- 
hensibility or reproducibility. 

Software-Distribution Methods 

When managing software, the integrity of appli- 
cation packages and the operating system ensure that 
all available applications are functioning. Knowing the 
origin of each file in the system is the basis to main- 
tain such integrity. Especially when software packages 
are installed intrusively, i.e., by copying them into the 
standard directory tree of a system, the tracking of file 
origins becomes difficult and thus needs to be exam- 
ined more closely. 


Fink and the Debian Packaging System 


Software built within the Fink Project [11] is 
packaged in Debian packages, that are installed with a 
well supported tool-suite. A database keeps track of 
the origin of installed files by associating them with 
their source package. An installer then relies on this 
database to ensure the package integrity. 


Software management using Debian packages is 
thus reproducible and comprehensible. In a larger 
environment where multiple versions of the same 
package need to exist file collisions can only be 
resolved through renaming or versioning, e.g., gcc-2.95 
and gcc-3.3. As not all files are easily relocatable and 
the resulting versioned structure always enforces ver- 
sion dependencies, creation and administration of such 
packages would be expensive. 


Network-based Distribution Via SEPP 
The SEPP-Packaging [12] system follows a dif- 
ferent concept to distribute applications. Rather than 
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installing applications file by file into an existing 
installation each SEPP-package encapsulates a ready- 
to-use application within a separate directory. Once 
this package directory has been copied or mounted 
over NFS the packaged applications are made accessi- 
ble to the user using stub scripts. By design SEPP sup- 
ports the coexistence of multiple versions even when 
dependencies on other SEPP-packages exist. 


Through its structure a SEPP installation is com- 
prehensible and yields reproducible results. In addition 
the Mac OS X standard proposes a similar approach to 
encapsulate the files of an application into a bundle 
directory. Thus incorporating Mac OS X support into 
SEPP is straight forward. 


A System Management Concept for Mac OS X 


Available Infrastructure 


When implementing our Mac OS X management 
concept, we relied on the existing infrastructure at our 
site. This infrastructure consists of the provided net- 
work services and the available tools. These two aspects 
of the infrastructure are discussed in this section. 


Available UNIX Network Services 


Centralization of services is a common approach 
to reduce redundancy of information. In the imple- 
mentation of the integration concept the following ser- 
vices were used: 

¢ DHCP: Management of network configuration 
and access control 

e LDAP: Consistent user authentication and 
group management 

¢ NFSv3: Network-based user homes and appli- 
cation packages 

e SMB: External access to user homes 


While all other services are only available using 
a single protocol to prevent confusion, user homes can 
be accessed using two different services: NFS and 
SMB. NFS is used as default for reasons of speed and 
flexibility. However the current implementation of 
NFS requires a trusted network and accordingly SMB 
is used for remote access from home or across 
untrusted networks. 


Chosen Tools 


Several administration tools were introduced ear- 
lier, each being particularly strong in one of the 
administration areas. This information is now used to 
determine the best suitable tools to implement an inte- 
grated concept: 

e Network Installation of Mac OS X: Jumpstart 
is the best choice even though reimplementa- 
tion is necessary because it runs fully automatic 
and completely unattended. 

¢ Configuring the System: The built-in limita- 
tion of complexity as well as the support for 
modularization make TeTre2 the preferred tool 
for a structured approach to integration. 
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¢ Serving Applications: Because the SEPP 
approach matches with the Mac OS X applica- 
tion concept SEPP is an ideal choice even 
though an extension for graphical applications 
needs to be implemented. 
In the next sections we first introduce each of the tools 
in its original form followed by a description of the 
changes necessary to make it work on Mac OS X. 


Network Installation of Mac OS X Based On UNIX 
Standard Protocols 


Design of Solaris Jumpstart 


Jumpstart is based on the standard UNIX ser- 
vices BOOTP/DHCP, TFTP and NFS. Once it is net- 
booted, it performs an unattended installation as 
described in Figure 6. 


The jumpstart system is net-booted on the target 
host. Therefore the DNS name of the target host is 
known to jumpstart at execution time and thus can be 
used to access a configuration directory on an NFS 
share with the name of the target host. This directory 
contains the pre-/post-installation scripts and a list of 
packages that shall be installed. 


The Reimplementation of Jumpstart for Mac OS X 


Our Mac OS X reimplementation of Solaris 
Jumpstart provides the same basic functionality and 
several useful extensions. Some of these extensions 
are listed in gray color in Figure 6. 

¢ DHCP: Although Darwin is compatible with 
UNIX standards, the net-boot implementation 
of Macintosh open-firmware is incompatible 
with the normal DHCP protocol and tries to 
start negotiations with the DHCP server. With a 
patch found in [13] it was possible to net-boot 
the installer using Linux and Sun Solaris. 
Override for hostname: One configuration file 
in TeTre2 associates all host names with their 
Ethernet MAC addresses. With the information 
from this file osxjumpstart can set the correct 
host name even when a host is booted of a tem- 
porary IP address and choose a different host 
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name for the configuration. We use this feature 
when installing portable Macs from a common 
set of “install only” IP addresses. 

OS Installation: Since applications are sup- 
plied using SEPP, only the operating system 
and extensions, e.g., fonts, are installed via 
osxjumpstart. The OS packages are provided in 
their native .pkg bundle form [4, 14]. Additional 
self-built packages are also provided in this for- 
mat for consistency. 

Software Update: Using Apple’s softwareup- 
date tool Mac OS X and installed packages are 
brought up-to-date. Unfortunately softwareup- 
date was designed to update an already running 
system. Thus the tool is run in a chrooted envi- 
ronment on the newly installed system without 
requiring a reboot. 

Management of Sensitive Data: All sensitive 
data, e.g., passwords, licenses, ..., is stored in an 
encrypted archive and the administrator is 
required to enter the encryption key at the begin- 
ning of the installation. In the post-install actions 
sensitive data is then copied into the system. 


Passwords for users in the netinfo database are 
stored in separate files under /var/db/shadow/ 
hash. The hash files can be copied to the ar- 
chive from any Mac OS X system with appro- 
priately set passwords. The same strategy can 
also be applied to the Auto-login password (/etc/ 
kcpassword) and the Open Firmware password 
(nvram security-password and nvram security-mode). 


Configuring the System 
Design of Template Tree 2 


Template Tree 2 (TeTre2) is a configuration man- 
agement system that structures the configuration of an 
entire site into modules — features in TeTre2 terminol- 
ogy. The configuration of a host is defined as a set of 
features whereas features consist of simple file opera- 
tions. Figure 7 shows the structure of TeTre2. Basic 
information like hardware type or Ethernet MAC 


¢ Determine IP using DHCP or BOOTP 


¢ Boot the installer system from TFTP, NFS 
e¢ Mount required NFS shares 


Config-retrieval 


© Get host name from IP-number and DNS 


e Override host name for specified Ethernet address 
e Use hostname to retrieve config info from NFS 


Pre-Install ¢ Format/partition disks 


OS/SW installation —@ Install required packages 
¢ Update the system using softwareupdate 


Post-Install ¢ Configure the system (TeTre2) 
Transfer sensitive data 
¢ Optional custom actions 


Reboot 





e Start the installed system 


Figure 6: Steps of a jumpstart installation (black) and the extensions for osxjumpstart (gray). 
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address of each host (the host.list file) on a site is sepa- 
rated from the individual host configuration for all 
managed hosts (file site.desc). 





Features 
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Figure 7: Structure of the TeTre2 Configuration: In 
the hostlist a hostname is associated with basic 
configuration information like installed OS, Eth- 
ernet Address or Network configuration that can 
be used as text substitutes in configuration 
actions. The site.desc file then selects the features 
for a host. Finally each feature contains a file con- 
taining a list of configuration actions that are 
applied to a host if the associated feature has been 
selected in site.desc. 


Features encapsulate a certain functionality, e.g., 
install and activate a service, or behavior, e.g., power 
management. The ability to enforce integrity allows 
flexible combination between features and avoids 
coincidental dependencies. Dependencies between 
features are designed by the administrator. As a conse- 
quence TeTre2 features obtain a semantic aspect in the 
context of system administration. 


Looking inside a feature one will find simple file 
operations such as copying files, creating and remov- 
ing directories and generating symlinks. The assembly 
of files from chunks, simple text substitution and oper- 
ating system dependent execution of file operations 
provide more flexibility for the configuration work. 
For the sake of comprehensibility TeTre2 never modi- 
fies the content of files. Either the files are fully con- 
trolled and consequently overwritten or the files are 
outside the scope of TeTre2 and thus ignored. Files 
with mixed content, e.g., that partially depend on 
TeTre2 configuration are managed by custom scripts 
running on the target system. For example allowing 
users to add their own printers and still manage a 
default set of printers through TeTre2 requires this 
mixed form of control. 


Overview Over the Mac OS X TeTre2 Features 


The functionality of the TeTre2 tool itself was 
sufficient to configure Mac OS X systems without 
modification. However a multitude of new features 
were required to control the diverse aspects of a Mac 
OS X installation. Figure 8 gives an overview of the 
implemented Mac OS X features. The categorization 
of the features is not part of TeTre2 but was intro- 
duced in the table for the benefit of the reader. 
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Basics: Automounter, Network 
configuration, Netinfo builder 
SSH public-key access, automatic 
package installer 


Admin tools: 


Services: Postfix, sshd, 
Filemaker/Meetingmaker server 
Clients: LDAP access, automount NFS 


homes, SEPP 
Default printers, custom 
loginwindow dialog 


User specific: 


Figure 8: Categories of TeTre2 features with examples. 


Unless documentation is available, features are 
created by identifying and verifying the differences 
that occur when settings are modified through the Mac 
OS X GUI. The concerned files are stored in the fea- 
ture and copied during the configuration process. If 
several features contribute to the content of a file, 
chunk-wise assembly is the solution. However when 
the content of a file depends on the current state of the 
system, e.g., hardware, the files are generated through 
scripts and the configuration for these scripts as well 
as the installation of the scripts are handled with 
TeTre2. Some of the features we added to TeTre2 for 
Mac OS X support were: 

e Package Installer Adapted from the correspond- 
ing UNIX feature an automatic post-install .pkg 
bundle installation is performed through a script. 
The package installer is run at each system start 
as StartupItem and is also run nightly by cron. 


Packages missing on the local system get in- 
stalled based on a configuration directory. Up- 
grades of existing packages and required reboots 
are handled, the latter is done through the reboot 
command or by sending an AppleEvent to the 
Finder when a user is logged in. 

¢ Netinfo Database Builder Netinfo data is 
stored in binary format making direct control 
through TeTre2 impossible. As the Netinfo data- 
base is required by Mac OS X for some applica- 
tions parts of Netinfo are regenerated at each 
start using the Netinfo Builder. Netinfo Builder 
is a Perl script that compares the current state of 
the Database with a desired state in the form of a 
raw dump (the term raw might be imprecise 
since the dump is ASCII but in a general for- 
mat). Netinfo Builder only changes the differing 
elements allowing a coexistence between man- 
aged and individual settings. 

Serving Mac OS X Applications from NFS 
Design of SEPP for Command-line Applications 


The design goal behind the SEPP [12] applica- 
tion distribution system is the encapsulation of ready- 
to-run application packages into individual directories. 
As a consequence the application executables inside a 
SEPP package need to be reconnected to the OS and 
its user. SEPP is structured into two parts that are 
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separated physically in two directory trees as shown in 
Figure 9. The application packages are located under 
lusr/pack/.... Optional support for NFS automount is 
built into SEPP allowing application packages to be 
assembled from various sources. 


The /usr/sepp/ directory is used to manage the 
applications on the local system making them available 
through the /bin/... directory. Rather than symlinking 
the executables from an application package, a Perl 
script — a stub — stands for the real application binary. 
Upon execution the Perl stub (in /usr/sepp/bin/) starts 
the real executable inside the package triggering the 
automount of the package containing the application. 
More precisely the Perl stub starts a package specific 
start script named startpl (inside each /usr/pack/. . .) 
which then can properly set everything up before start- 
ing the application. The concentration of all adminis- 
tration information into one directory (/usr/sepp/) makes 
it possible to distribute this directory via NFS resulting 
in an application system that is fully centralized. 


SEPP for Mac OS X 


SEPP supports Darwin and X11 applications on 
Mac OS X in the unmodified form. In the case of 
Cocoa or Carbon applications the basic structure of 
Mac OS X applications [15] comes into play: Mac OS 
X applications are packaged into a single, structured 
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app bundle directory conteining application data or 
additional libraries required ‘vy the application. Usually 
the files and directories with n the bundle are addressed 
by relative paths allowing an .app bundle to be moved 
around on the system without re-installation. 


Making the packaged <pplications available to the 
Finder is more of a challeage. A closer look at the 
anatomy of a .app bundle however leads to the following 
solution. The Finder only needs a part of the bundle files 
to display the application and associated documents. We 
use this fact in the Mac OS. X extensions of SEPP by 
copying all the required filzs to form an independent 
miniature application — an application stub — in the 
/usr/sepp/macosx/ directory. In this application stub a small 
Perl script is used to start th real application. Figure 10 
shows the structure of such ain application stub directory. 
By making the contents of the /usr/sepp/macosx directory 
available as an item in the I‘inder’s left hand navigation 
bar users can access all SEPP applications in a natural 
Mac-like way. 

Special Application Support 


All binaries in a SEPP package are started indi- 
rectly through the package specific start.p! script. One 
of the advantages of such i script is the ability to pre- 
pare the environment for an application. Mac OS X 
SEPP packages make regu ar use of start.p| since many 
applications do not follow the standard exactly. 


a 


pack 


i 


mypkg_a myock_b 


a 


>» start.pl 


[at] 


ape Stars.pl 


4, 


Figure 9: Structure of the SEPP system: The original SEPP system designed for command-line applications is 
shown in black and the extensions implemented to support Mac OS X GUI applicztions are shown in gray. The 
boxes in the figure represent executables. Rounded boxes are used for the Perl stubs that a user will execute 
directly while the cornered boxes stand for the real applications. The individual start.pl files found in every pack- 
age provide customization for the application to be started. The dotted arrows shcw the chain of actions when 


an application is launched. 


.../Sample.app/ 
Contents/ 
Info.plist 
MacOS/ 
the_executable 
Resources/ 
sample_icon1.icns Displayed in Finder 
sample_icon2.icns 


Contains filenames of executable and icons 


Perl script replacing the real executable 


.. CNS All other Icons referenced in Info.plist 


Figure 10: Structure of a Mini application bundle used as application stub. Entry poiat is the Info.plist file in which 


all other filenames of the bundle are found. 
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ABSTRACT 


System administrators are faced with the challenge of managing and compiling information 
about deployed systems to ensure the maintenance, scalability, security, and overall availability of the 
infrastructure systems. Recently, securing and monitoring the deployed enterprise infrastructure 
systems has taken unprecedented importance due to the added accountability now being placed on 
managing corporate data and private information. Compliance with new regulation has never been 
stressed more than now — with SOX/Bill 198, GLB Act/PIPDEA, HIPPA to name a few regulations. 


Toward this, we present RegColl: A practical framework for registry and system configuration 
management for monitoring and maintaining the health of corporate infrastructure systems in Windows 
environment. In RegColl, registry and configuration information are collected at a centralized location 
so as to inspect and analyze the data for policy compliance, system configuration monitoring, and 


incident response services. 


Introduction 


Managing hardware, operating systems, and 
applications on workstations and desktops in corporate 
and large networked environments is a big challenge, 
one that System administrators deal with on a day to 
day basis. To monitor the health of infrastructure sys- 
tems, it is imperative to understand system behavior, 
especially aspects that are related to the upkeep of the 
system and eventually the whole network. Windows 
registry provides system information to understand the 
behavior of the deployed applications. 


Monitoring the system’s registry and configura- 
tion information offers important insights in deployed 
software, patches, drivers, etc. — such entries can be 
analyzed for vulnerabilities, troubleshooting, and com- 
pliance with standards, along with providing docu- 
mentation in an event that might have reporting impli- 
cations. Further, by using the registry and administra- 
tive tools in Windows, system administrators can pro- 
vide local or remote support, query registry logs for 
security analysis, and can apply compliance checks. 
We extend this facility in centralizing system monitor- 
ing services to run audits, apply compliance standard 
checks, and enforce corporate & legislature policies. 


The problem of maintaining healthy configurations 
of a large installed base and other third party software 
packages has been recognized as a daunting task [10]. 
The growing numbers of software and changing system 
configuration makes it even more difficult to specify an 
“ideal configuration,” which makes troubleshooting and 
maintenance problems intractable [8]. Aside from com- 
pliance and network monitoring issues, system trou- 
bleshooting expenses drive the total cost of network 
ownership, with maintenance and support costs. Largely, 


troubleshooting cases are due to system misconfigura- 
tion [8] and hence, it is one of the issues we propose to 
tackle with RegColl’s centralized registry framework. 


Non-compliance with legislature requirements 
such as Sarbanes-Oxley [16], Gramm-Leach-Bailey 
(GLB) Act [9], Health Insurance Portability and 
Accountability Act (HIPAA) of 1996 [11], and others 
that preserve private information is putting more and 
more organizations under scrutiny. Such compliance 
deviations are not only a breach of law but they also 
affect client confidence. Further, legislatures like OCC 
(Office of Comptroller of the Currency) [4] and 
HIPAA require organizations to maintain an informa- 
tion security program that includes “Detecting, Pre- 
venting and Responding to Attacks, Intrusions, or 
other Systems Failures.” 


Computer intrusions [5] leading to theft of intel- 
lectual property or any system compromise in a corpo- 
rate network can jeopardize users’ and employees’ 
data integrity. Furthermore, any incident leading to 
misuse of confidential data can put a company’s repu- 
tation at risk. Implication of such incidents may 
include comprehensive incident response plans and 
liability to notify the reporting agencies and the 
affected customers. 


Traditionally management plans are devised for 
Network and IT Infrastructure which cover the secu- 
rity policies, audits, asset inventory, and incident 
response. Such plans ensure business continuity and 
disaster recovery in the event of computer and net- 
work intrusion or system compromise. Federal Trade 
Commission (FTC) Safeguards rule [7], OCC and 
Medical Privacy rules like HIPAA mandate an inci- 
dent response plan which encompasses ways to con- 
firm whether an incident occurred and provide accu- 
rate, relevant, and timely information. 
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We present the benefits of a centralized registry 
collection and monitoring approach in the next section. 
Subsequently, we give the architecture details of Reg- 
Coll framework and its components including the col- 
lection mechanism and transfer mechanisms, and we 
discuss the implementation of RegColl framework. The 
related work section compares some of the existing 
technologies and their goals with RegColl framework. 
We then present the future work section, showing the 
ongoing development efforts and the proposed 
upgrades of RegColl framework. We then conclude, 
revisiting the usage and presenting the practical solu- 
tions that might be deployed to realize some of the key 
benefits of RegColl. 


Why Registry Monitoring 


To address requirements of system monitoring, 
configuration management, and incident response, we 
take a proactive system management. In this approach, 
we freeze configuration states that can stand compari- 
son to a misconfigured system. In the event of an inci- 
dent these frozen states provide a historic log of sys- 
tem registries containing restore points that allow for 
reconfiguration, System rebuild, and incident response. 


The centralized registry offers key benefits like 
Policy enforcement at a centralized location, Audits 
and Analysis of configuration information, and the 
overall System Configuration Management. The fol- 
lowing sections elaborate these benefits. 


Policy Enforcement 


Corporate and legislature requirements mandate 
formulation of a management plan and steps to control 
access to infrastructure systems, and ensure the secu- 
rity of financial data and non-public information. Cen- 
tralized repository serves these purposes by providing 
a single point of policy validation and enforcement. 
Audits and compliance requirement checks are 
achieved by deploying compliance and auditing stan- 
dards on the stored configuration. 

¢ Remediation — Centralized configuration offers 
remediation capabilities to easily and effectively 
automate and monitor policy consistency 
requirements. The remediation technique allows 
for automated and user controlled repair process, 
for example, re-importing registries to enforce 
corporate policy across thousands of machines. 

¢ Restore Points and Time Travel — Centralized 

configuration management allows for system 
restore and time travel by maintaining restore- 
histories. The causal relationship between the 
change information ensures a mechanism identi- 
fying the restore points and version changes. 
Such a tracking mechanism would enable reinte- 
gration of changes when a disconnected entity 
rejoins the network. Restore points and configu- 
ration change tracking features facilitate rollback 
to a stable state system state in the event of soft- 
ware misconfiguration. 
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¢ Change Management — Centralized configura- 
tion management detects and responds to 
planned and unplanned changes by setting stan- 
dards and policies for change enforcement. 
Registry and configuration information cap- 
tures the changes in system’s configuration; any 
planned changes can be verified by studying 
the configuration change information. 


Audit & Analysis 


Risk management practices require audits and 
analysis of configuration information to assess and 
mitigate any risk arising from vulnerabilities in the 
system. Analysis of system, registry, and configuration 
logs is an important aspect of troubleshooting. Further- 
more, such techniques take more prominence in rating 
and certification of processes and organization, and 
not ignoring that they are legislature requirements as 
well. Towards this, a centralized repository will serve 
as a common source for audit and analysis such as: 

¢ Security Auditing — The RegColl framework 
proposes to collect and compare thousands of 
configuration elements from a networked sys- 
tem to identify and enforce security configura- 
tions standards. For example, security rele- 
vancy check [18] is done by inspecting security 
relevant registry keys. 

¢ Dynamic Auditing — Centralizing registry and 
system configurations for dynamic auditing 
allows for systems to be audited seamlessly at a 
central and secured location on a regular basis. 
Auditing ensures consistent compliance and 
provides detailed reporting to address regula- 
tory compliance requirements. 

e Security Analysis — Centralizing registries 
give a single source of security and vulnerabil- 
ity analysis for tools developed and discussed 
by Wang [8] & Wenliang Du [18]. 

¢ Gap Analysis — These stores prove to be a sin- 
gle source for running gap analysis along with 
assessment of system configurations for known 
configuration vulnerabilities including patch 
inconsistencies. 


System Configuration Management 


Registries and configuration information is 
shared and accessed by multiple applications, such as 
drivers, software, and patches — it is imperative to 
cache a snapshot of this information so as to establish 
checkpoints at every change. Registry changes are 
monitored and pushed to a centralized location as and 
when new changes are recorded. The monitoring 
process polls the registry and configuration files and 
triggers the export of registry database as key entries 
are changed in the registry. These system-state snap- 
shots are stored into a centralized secure location to 
analyze system vulnerabilities, run audits, enforce cor- 
porate and compliance policy checks, and monitor for 
any rogue software installations. 
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Incident Response 


Incident response identifies ways to confirm 
whether an incident occurred and then provide accu- 
rate, relevant, and timely information. In such an 
event, the incident information gives implementation 
controls to secure the systems and crime scene to pro- 
tect individual rights established by policy and law. 
Towards such requirements, a centralized registry and 
configuration repository provides a clean backup of all 
registry and configuration changes as they happen. 


Corporate Data Protection 


Protecting corporate data such as customer infor- 
mation is part of regulations, including the statewide 
California Database Security Breach Act, the Gramm- 
Leach-Bliley Act for financial services firms, and 
HIPAA for health care providers. These regulations 
are factors that drive corporations to beef up efforts to 
prevent unauthorized disclosure of sensitive data. In 
the recent past, several corporations, including large 
banks (Bank of America and Wachovia), have 
acknowledged substantial data leaks affecting tens of 
thousands of customers. Evidently, such leaks origi- 
nate from inside the corporate office itself, often with 
data being transferred to or from a range of peripherals 
such as zip drives, CD/DVD drives and portable print- 
ers, to name a few. Furthermore, monitoring system 
ports and other peripheral devices can be used to block 
the entrance of viruses and mal-ware, and prevent cor- 
porate data from being copied or saved except where 
and when authorized. RegColl runs on all nodes on all 
the network clients and interrogates the Registry of 
every system attached to the corporate LAN. RegColl 
analyzes the registry’s record of activity to identify 
what interfaces are being used, what peripherals are 
being attached to the system, and which of those 
devices are currently active. 


RegColl’s ability to examine peripheral devices 
connected to system allows for enforcement of data 
protection policies. For example, defining a corporate 
data protection policy, which makes it illegal to copy 
or move data to portable devices, can be tracked by 
running the policy enforcement tools. RegColl is not 
designed to monitor and capture use of peripheral 
devices; however, it can extend the functionality to 
incorporate the configuration changes from hardware 
driver installations. 

The next section delineates architectural compo- 
nents of RegColl’s framework. 


Architecture 


The managed entities in this system are worksta- 
tions, desktop computers, and laptop systems. A 
process, called RegistryMonitor, runs on the managed 
entities to collect the registry and configuration infor- 
mation changes. The following sections give the 
details of each of the components and their interac- 
tions. We introduce the version and causal relationship 


RegColl: Centralized Registry Framework ... 


mechanism of the change sets, the collection points, 
and the collection server components. 


Architecture Overview 


The RegColl framework introduces a registry and 
configuration information dissemination mechanism 
supported with a centralized repository. To ensure bet- 
ter availability of this information and protect against a 
single point of failure, RegColl introduces a secondary 
fail over server, which is accessible to the monitor 
process in the event of primary server failure. 


RegColl takes into account the disconnected 
operations. We assume that some managed entities 
may enter disconnected mode when mobile. In such an 
event the registry monitor process collates the infor- 
mation locally with a causal version log. This local 
storage ensures that no critical changes are disre- 
garded or missed when disconnected. Since RegColl 
works with causally related changes, the registry and 
configuration changes are reintegrated with the previ- 
ously collected versions. 


Since RegColl’s framework collects and stores 
system information in a chronological manner, this ar- 
chive reflects restore points that can be retrieved based 
on a version identifier. The causality allows for time 
travel and undo capabilities. Maintaining such a ver- 
sion relationship ensures that a disconnected entity 
when joining the network can seamlessly integrate the 
collated change information with the repository. The 
following subsection provides a systemic view of the 
RegColl’s components. 


Collection Points 


The registry and system configuration information 
is collected by the registry-monitor process deployed as 
part of the registry collection framework. The registry- 
monitor acts as a collection source on each of the infra- 
structure systems’ managed entity to gather registry 
and system configuration changes. These updates are 
cached when the registry keys are modified or a new 
sub-key is inserted or when a new node is created. The 
centralized framework identifies the keys that are most 
often updated and reflect critical modifications. 


Registry and system configuration files tend to 
grow with time and configuration changes, such as 
updating new patches for operating system, security 
updates and driver installation. These registry logs 
might swell from a few kilobytes in size to multi 
megabytes (between 20 KB and 600 MB) as time 
elapses and configuration changes are made. To over- 
come the problem of caching and transporting heavy 
files, RegColl uses a differencer to identify changed 
entries and transport these deltas [1] to a registry col- 
lection server. 


Often computer systems in a network, especially 
those in corporate networks, may be disconnected for 
mobility. Such a scenario mandates local caching of 
registry change snapshots (referred to as change-sets 
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in RegColl framework). To prevent the collection 
point of a disconnected system to make failed attempts 
to contact the collection server, the cache is polled 
before each connection attempt to the collection 
server. After the system is reconnected, any pending 
change-sets are pushed before a new snapshot is gen- 
erated (Figure 1). 


In order to actively observe the registry changes, 
monitors that have an operating system hook can pro- 
vide most efficient cues on the registry changes. Reg- 
Coll incorporates Microsoft’s FileSystemWatcher 
component to notify (callback) Collection Server in 
the event of any registry file changes. Since registry 
information is critical, it is by default authenticated 
and encrypted using symmetric key cipher and hosted 
in the registry store. 


Collection Server 


The registry collector (primary server) acts as a 
collection point for registry and system configuration 
file entries. These entries are archived in the history 
server with version information, and time stamped for 
chronological ordering. As mentioned, the version and 
ordering information gives undo and time travel capa- 
bilities, since causally related entries can be queried 
for a particular version. The Collection Server ar- 
chives the restore point for each system and maintains 
a mapping between the monitored system ID and the 
collected configuration files. The fail-over server 
architecture offers robustness and reliability to the 
overall framework. Exported registry entries are 
hosted on the primary registry server; in an event of 
server outage, the secondary server acts as backup. 
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The primary and secondary servers synchronize main- 
taining the consistency in registry updates. This data 
consistency and synchronization is controlled using a 
causal history based replication framework [3]. 


Configuration and Registry Repository 


Configuration and Registry repository provides a 
uniform monitoring infrastructure and is the single 
source of information necessary to develop the best 
practices for change, problem, and incident manage- 
ment. This source of archived registry & configuration 
log helps in configuration discovery & dependency 
mapping, and change tracking serves as the trusted 
information base for IT operations improvement. 


The principal goal of configuration and registry 
repository is to tag the change-set version and store 
them in a multi-mapped manner while preserving their 
chronological order. This repository provides an accu- 
rate, always up-to-date configuration repository of the 
servers, infrastructure software, and dependencies 
between registries. Centralized registry enables system 
monitoring services and is a definitive source of auto- 
mated, application and infrastructure configuration 
information. 


Compliance and Analysis 


A registry repository’s ultimate goal is to serve as 
a single point of network monitoring service. The con- 
structive contribution of a centralized registry and con- 
figuration store would depend on what tools are 
employed to query the repository and how frequently 
the repository is checked for compliance. The fre- 
quency of compliance check for policy compliance 
determines the effectiveness of centralizing the registry. 
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Figure 1: RegColl architecture. 
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Compliance checks at regular intervals would be 
able to respond to compliance deviation more effec- 
tively. However, a compliance check in the passive 
mode may be constantly looking for a pattern of mali- 
cious behavior. For example, if a policy deviation or 
deficiency is found, the system may trigger a host of 
alerts to isolate the host immediately until the host has 
been remediated. A host of third party software [17] 
can be employed to check the legislature requirements 
and perform system analysis. 


Security Consideration in RegColl 


The centralized registry framework requires enti- 
ties to establish their identities in the form of public 
and private key pair. The creation of public-private 
key pair for each machine is performed as a system 
registration step. Registry logs that are collected as 
change sets (at collection points) are signed by the 
system/node’s private key, which are authenticated to 
enable uploading to the collection server. Signing 
change sets between the collection-point (Registry 
Monitors) and the collection server (Registry Collec- 
tor) establishes confidentiality and proper usage. 


The framework enables trust in collection enti- 
ties by certificate exchange. Entities are governed by 
their identity keys (private keys); hence, registry infor- 
mation shared is secured from external attacks and 
malicious updates. Such a trust-framework enables 
revoking a key in the event of a malicious change-set 
being generated. 


Another important security aspect is the user’s 
system privacy and invasiveness of remote monitoring 
tools. We contend that our framework is less invasive, 
since the differencer tool compares and generates 
deltas for each new registry change. Change set deltas 
enables RegColl’s registry monitoring service to con- 
sume orders of magnitude less bandwidth than any 
other active monitoring process. Further, remote man- 
agement and monitoring tools (like, Tivoli and Open- 
View) allow for exclusive control of the system, 
wherein RegColl framework’s registry snapshots 
always has read access only. The system administrator 
will never have access to any other file, hence preserv- 
ing user’s privacy. 

Registry and Configuration Change Monitoring 


We explain the registry and configuration 
change-set generation mechanism in this section. The 
change capturing process requires a policy to deter- 
mine the frequency of changes. We share our experi- 
ence in determining these policies along with a discus- 
sion on Operating System tools used for capturing reg- 
istry and configuration changes. 


Registry change generation synchronizes the 
local registry changes of managed entities with the 
remote collection server. These managed entities host 
a registry monitoring tool called RegistryMonitor, 
which publishes the change-data from the managed 
entities based on an “‘auto-publish”’ policy. 
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The publish operation policy determines how 
often the registry changes should be published or what 
should trigger the synchronization. In RegColl’s 
implementation we reviewed publish policies based on 
“elapsed time” and “transaction size” of the registry 
updates. Elapsed time mechanism assumes changes to 
be comparatively smaller if the time units are main- 
tained small enough. This proved to work fine until 
the user tries to install large software. For example, 
installation of Oracle database induces numerous reg- 
istry changes. We found that large installation gener- 
ates so much traffic that it affects the efficiency of 
change-set generation process. Hence, we recommend 
fine-tuning the change-set generation and publish poli- 
cies in the event of planned large installations. 


In the implementation scenario, another chal- 
lenge is determining the publish policy. The policy 
should be set for smaller transaction size or smaller 
time interval, which inversely affects the change-set 
size. Too fine granular publishing affects the perfor- 
mance and usage of the system. On the other hand too 
coarse grained or too many updates are found to pro- 
duce very large change-set, which are detrimental in 
exchange operation and hence, proves to be an ineffi- 
cient use of server resources. 

Our experience in determining the publish policy 
comes from monitoring native file system changes 
using FileSystemWatcher component on Windows 
[13] and Inotify file system event monitoring mecha- 
nism [6] for Linux. FileSystemWatcher, of Microsoft 
.NET framework’s System.IO namespace gives access 
to system functionalities such as the FileWatcher util- 
ity. The FileWatcher listens to the file system change 
notifications and raises events when a directory or file 
in a directory changes. This facility is utilized to mon- 
itor registry-file changes. 


Differentiating between users initiated change 
and OS process changes are not discernible to the file 
watcher. For example, interaction between Windows 
Explorer (explorer.exe) and user workspace are 
reported as file changes. Further, any file creation 
leads to multiple file change notification. 


On a Linux based system, a number of swap files 
are created when user programs open a file. Our expe- 
rience shows that these files are created and deleted 
within a few milliseconds. Capturing these temporary 
files as part of change-set description proves to be 
unwanted overhead, both in terms of network traffic 
and change set log size. To manage such copious 
change notifications, we analyzed both FileWatcher 
and Inotify change log pattern and developed a filter 
routine that prunes junk and redundant entries. 


RegColl follows a transactional semantics where 
all changes take place atomically. A time based publish 
policy will divide the transaction into multiple units 
should the installation triggered change time is greater 
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than that of publish time. For example, software like 
Tomcat web server has a longer installation and con- 
figuration time than the default publish time. We ran 
our experiments on University network configured to 
trigger notification in the events of file creation, file 
access and write, file name and directory name change. 
The size and security attributes changes are also 
reported. We found that the file system watcher com- 
ponent has certain limitations, especially in reporting 
what process initiated the registry-file changes. 


We contend that monitoring only the registry-file 
changes will improve the overall performance of syn- 
chronizing the configuration files since only the registry 
changes are propagated to the server reducing the over- 
all network traffic. Even with snapshotting technique, 
the snapshot size can be prohibitively expensive given 
that changes happen in number of configuration files; 
on the other hand, RegColl captures specific registry 
and configuration changes in a given time window. 


Based on our experience in analyzing change noti- 
fications, we devised the staging policy where file 
change notifications are collected for 30 seconds and 
flushed in an event if no new change is detected in this 
time window, i.e., the inactivity window. An alternative 
scheme for pushing the change-set is if the queue reaches 
a predetermined size (e.g., 100 numbers of changes in 
our case). Nonetheless the publish policy is configurable. 





BRO CURA no 


Peete ORs e Tub eaey ay 
2; yvegnmonitor [-options] 
where options include: 


“login <usy> <pwd> 


authenticates 


Kang, Sharma, and Thanki 


RegColl Implementation 


The Windows registry is a hierarchical structure 
of key and name value pairs. This structure acts as a 
central configuration database for users, applications, 
and other system information. Each key in the registry 
is a node in the hierarchical structure with one or more 
sub keys associated with it. This structure binds 
together configuration information in the name-value 
pair container associated with each node. 


A “registry key” is of importance if it can reflect 
on configuration changes. If such keys and other sys- 
tem configuration information are collected at one 
location, it provides a centralized source of registry 
and configuration information that can be used for 
system auditing and analysis. To realize this single 
source of system information repository the following 
section gives the implementation details of the pro- 
posed framework. 


Registry Monitor 


The centralized registry collection framework 
requires network wide deployment. The current imple- 
mentation bundles the collection point interface as 
Registry monitor batch file, regmon.jar, and reg- 
mon.properties files. RegistryMonitor batch file runs 
the Registry Monitor on the user’s local workstation; 
regmon.jar has the APIs to identify registry changes 


user on collection server 


~logout 

cee A SD 

~nonitor 

~mnonitorpath <file path> 
—nonitorall <file path> 
~showlog 

~verbose 

~server 

—help 

“exit | ~quit 


invalidates session and logs out 

starts registry monitor 

monitors default registry and systen configuration file 
monitors configuration file specified 

monitors default registry and specified configuration File 
show registry change log 

enable verbose node 

show current collection server session 

show usage 

exit systen 





Figure 2: Registry Monitor interface options. 








>NWINDOWS Ns ystem32\conf ig NDEFAULT 

Pee Gs ile) See is ott ue Peete ie ae ke esd by 

=\WENDOWSN\s ystem32\conf igNSECURITY 
=\WINDOWS Ns ystem32\conf ig NS OFTWARE 
>\WINDOWS\s ystem32\conf igNSY¥YSTEM 

=\Documents and Settings\pgthankiN\NIUSER-.DAT 





Figure 3: Registry Monitor with “-monitor” command. 
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and communication protocol implementation for col- 
lection-point and collection server communication for 
pushing changes. The regmon.properties file has col- 
lection server information, i.e., collection server’s 
fully qualified domain name (FQDN) and port on 
which the server is listening. 


To fully understand the collection framework, we 
deployed and tested the system on a local Windows 
network. The following section gives a deployment 
scenario of registry monitor and registry collector. 


Figure 2 shows the screen shot of Registry Moni- 
tor implementation. RegistryMonitor monitors the 
default registry files; DEFAULT, SAM, SECURITY, 
SOFTWARE, SYSTEM stored in the windows\sys- 
tem32\config directory and NTUSER.DAT stored in the 
user home directory. Files being monitored could be 
seen by invoking the “monitor” command, as shown 
in Figure 3. 

Installation of any new application on Windows 
operating system essentially modifies four registry keys, 
ie, HKCR (Classes Root Handle Key), HKLM (Local 
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We tested Registry monitor by installing and 
uninstalling a few software and drivers. For example, 
as shown in Figure 5 and Figure 6, Registry monitor 
when started with the “-start”” command has success- 
fully identified a change in NTUSER.DAT registry file. 


Figure 6 shows the registry modification after the 
user installed the ‘Mozilla FireFox’ application. In the 
event of Registry monitor polling the registry, the 
FireFox application installation will be identified as a 
registry change. Registry monitor’s file differencer 
logs the change in the change set as shown in Figure 6. 
Registry Monitor Setup 

Registry monitor setup entails setting up a moni- 
tor code at each of the infrastructure system (what we 
refer to as, collection point). Each deployment requires 
setting up of public and private key pairs at the collec- 
tion point. These keys secure the communication chan- 
nel between collection points and collection servers, all 
registry change sets are signed with collection point’s 
private key. The registration process requires the sys- 
tem administrator to authenticate themselves before 
enabling deployment process. Users set up a secured 


Machine Handle Key), and HKU (Users Handle key). 


password for client log on process. 


=Nvegcollisrco>regqnonitor —login —-usr pathanki —pwd 


login successful 

ou are logged in as pgthanki 
urrent time is 

our session id is 

our session token is valid until 


28605-@5-12 22:14:35.283 
EEG hgh 
v4 


4:35.283 


Currently logged on to Windows Networking as “pgthanki” 


Status: 


Registry Monitoring Service is running normally 





Gurrently logged on to Windows Networking as “pgthanki" 


Status: 


Registry Monitoring Service is running normally 


Identified changes in "C:\Documents and Settings\pgthankiN\NIUSER. DAT 
Current time: Thu May 12 22:97:19 EDT 2685 


Pushing changes on to collection server: 


Operation Successful. 


COIT—PGTHANKI .uncc.edu 





Figure 5: Identified Registry Changes using Registry Monitor. 
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Registry monitor requires users to authenticate 
with the collection server before initiating the registry 
monitor process. The logon process is session bound 
(Figure 4) and each login returns a session token from 
the collection server, which is usually valid for a 
period of 24 hours. The RegistryMonitor interface 
manages the session ID locally by encrypting it using 
a pre-established symmetric encryption key (set up 
during the machine registration process). User can 
logout of the system by calling “‘-logout”’ to invalidate 
the session token. The session token and session IDs 
are discarded once invalidated. 


Registry Collector 


The registry collector server maintains a direc- 
tory structure of registry logs for each collection point 
(Figure 7). The registry collector captures the change- 
set snapshots, which are incremental delta information 
supplied by the collection point to reflect the registry 
changes. This contains the size of exchanged data at a 
given time since only the delta of change-sets are 
moved and not the whole registry or system configura- 
tion file. Moving an entire registry snapshot (taken at 
the time of registry change) is a process and network 
intensive operation. Hence, to avoid loading the net- 
work or system resources, only the change set deltas 
are generated and uploaded to the collection server. 


These deltas give an incremental version change to 
the registry collection. Since the information traveling on 
the network is a delta of change set (or simply the changed 
data blocks), the information exchange is lighter and faster. 


The registry collector maintains the version order- 
ing for each change-set supplied by the collection points. 
These entries are multi mapped with MachineID, collec- 
tion system’s ID and the timestamp they are associated 
with. Figure-7 shows the collection servers format of 
storing change-set deltas. 
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Related Work 


Monitoring system configuration information has 
been employed in many systems. The following are 
some examples. While these tools share the same idea 
of utilizing system configuration information as Reg- 
Coll, they differ in the collection mechanism, such as 
disconnected mode support and its usages. RegColl 
collates the system configuration changes into a cen- 
tral location so that the centralized system configura- 
tion (e.g., registry) repository could incorporate the 
useful infrastructure system monitoring services such 
as compliance checks, incident response endorsement, 
and corporate policy validation. Similarly, Windows 
XP system restore [12] can collect registry data in its 
state snapshots. 


The STRIDER [20] project uses differencing of 
periodic snapshots to reveal any configuration changes 
in the Windows registry. The registry keys of a failing 
program are monitored and recorded for analysis, this 
helps examine and reflect on fault in the program. 


UNIX-based tools like Chronus [2] detect con- 
figuration error that might induce a faulty state in the 
system. Chronus captures the failure, inducing state 
changes to differentiate between working and non- 
working states. By using binary search, it can diag- 
nose a range of common configuration errors for both 
client-side and server-side applications. It helps to 
reveal the specific failure cause, enabling recovery 
with minimal lost state. Another UNIX-based tool for 
system change monitoring is discussed in Backtracker 
[15]. Backtracker uses a change log mechanism and 
maintains an operating system causal history log. 
These logs are analyzed to determine the configuration 
changes which might be caused by the installation of 
an application or a computer intrusion. 
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Figure 6: Changes identified by Registry Monitor. 
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Future Work 


RegColl is evolving rapidly and the skeleton is in 
place. We would like to invite users to try this tool, 
which can be extended to collect other system info 
files such as the .ini files on Windows and .rp resource 
files on UNIX systems. The tool is ready for down- 
load; please send an email to bbkang@uncc.edu for 
registry monitor bundle and/or any further information. 
This bundle, along with other resources can also be 
downloaded from the following link: http://coitweb. 
uncc.edu/bbkang/ISR/. 


Conclusion 


The RegColl framework is positioned as a back 
end collection entity that seamlessly collates the reg- 
istry and system configuration changes. A fail over 
server adds reliability to the architecture and provides 
a backup for change information. Deployment of such 
a server would obviously be on the network but away 
from the internet. Further, configuration servers are 
patently isolated and kept secure. However, we 
employ security considerations that mandate setting up 
public & private key pair at each collection point. All 
change sets are signed by the machines private key. 


We delineate some of the useful contributions of 
system monitoring services that utilize the RegColl 
framework. 


Corporate Policy Enforcement 


The collection server can be deemed as a policy 
enforcement point, where policies are monitored and 
enforced by third party tools. This central location gives 
a validation source for such tools; if there is a policy 
deviation, a policy enforcer tool will generate an alert to 
the system admin. Audit and configuration are just log- 
ging tools, and a configuration management tool will 
identify if the registry is in good shape based on which 
rules the system administrator decides to apply. For 
example, Wang, et al, proposed a tool for troubleshooting 
misconfigured systems [8] using registry information. 


Incident Response 


Centralizing registry and configuration informa- 
tion meets the preliminary incident response require- 
ments of documenting and confirming an incident. If 
an intrusion or system compromise is detected, it will 
be useful to analyze the changes that happened in the 
course of intrusion. The incremental change informa- 
tion is constructive in pinpointing specific changes. A 
compromised system’s registry and system configura- 
tion information can be compared with a previously 
identified valid state. 


Non-Invasive Monitoring 


Active scans of a system’s configuration and reg- 
istry information can be considered more invasive 
process than capturing registry snapshots. The CPU 
and network usage of an active monitoring system 
(e.g., IBM Tivoli) consume order of magnitude more 
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resources than a prescheduled maintenance task or 
remote monitoring. 


Capturing registry information on a timely basis 
makes registry monitoring a lot less invasive than 
remote management, since the “differencer” produces 
deltas for each new change in the registry. Registry 
Monitor’s monitoring service consumes orders of 
magnitude lower bandwidth than an active monitoring 
process. Remote management allows full control of 
the system where system configuration changes are 
accessible for read/write operations remotely. In com- 
parison, the RegColl framework’s registry snapshots 
are read-only. The collection server’s administrator 
never has access to any other files, thereby preserving 
the user’s privacy. The system configuration informa- 
tion is stored in an encrypted form which adds to the 
overall information protection. 


The single source for auditing, analysis, and pol- 
icy validations capabilities gives more control and 
monitoring power to system administrators managing 
large corporate networks and infrastructure systems. 
Hence, we believe RegColl’s centralized registry and 
configuration framework will be a useful tool in over- 
all infrastructure systems management. 
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ABSTRACT 


Laptops running UNIX operating systems are gaining market share. This leads to more 


sysadmins being asked to support these systems. 


This paper describes the major technical decisions reached to facilitate the pilot roll out of a 
mobile Mac OS X platform. We explain which choices we made and why. We also list the main 
problems we encountered and how they were tackled. 


We show why in the environment of a customer support organization at Cisco, a file 
management tool with tripwire-like capabilities is needed. Radmind appears to be the only 
software base meeting these requirements. We show how the radmind suite can be extended and 
integrated to suit mobile clients in an enterprise environment. 


We provide solutions to automate asset tracking and the maintenance of encrypted home 
directory disk images. We draw conclusions and list the lessons learnt. 


Introduction 


Mobile UNIX Platforms 


People have been using UNIX for more than 25 
years. For much of that time computers running UNIX 
were large, clunky machines that could only be called 
mobile if you happened to own a truck. The physical 
location of the computer and its network address 
remained the same for much of the machine’s lifetime. 


Today we are faced with mass-produced UNIX 
laptops that move around the world as fast as their 
users do.1 


Solutions specific to UNIX machines need to be 
put in place to handle UNIX laptop deployments in the 
enterprise. 


About Us 


Cisco’s [2] Technical Assistance Center (TAC) is 
a worldwide technical support organization. The TAC 
provides technical support to Cisco customers. We, the 
authors of this paper are members of the IT support 
group for the technical support engineers of the TAC. 


All development, testing and debugging of net- 
working products at Cisco is done on UNIX platforms 
— mostly Suns [5] running Solaris. Engineers need 
UNIX because of the large amount of utilities (some 
of them internal) available only for UNIX platforms. 


In the last few years some of our users had been 
given Windows laptops, many of them had installed 
Cisco’s version of Linux [3] on them in order to get 
access to the UNIX tools they needed. Recently our 
support group noticed a shift towards Apple [1] 
PowerBooks. It became clear that a supported mobile 
platform with UNIX capabilities was something our 
user base would greatly appreciate. 


1Some laptops move faster than their users due to incorrect 
baggage handling or theft. 


Additional Material 


You can find additional material at http://rad- 
mind.org/contrib/LISA0S5 . There you’ll find: 

e The radmind extensions discussed in this paper 
(distributed as patches) 

e The scripts called by cron to update the system 
(radmind-update) and some scripts that toggle 
radmind-update features 

e The login scripts enforcing the use of FileVault 

e The asset database code (client and server) 

e Various tidbits and scripts that could be of use 
to other system administrators 

¢ MCX configuration examples 


Choices 


Constraints 


The following constraints are imposed on us by the 
department using our services: 
¢ Our environment needs to be highly available 
(even the desktops). We can meet this require- 
ment by keeping spare desktops around, clus- 
tering our servers and distributing our services. 
e Users need access to UNIX tools 
e Users need a mobile platform so they can work 
from home or while visiting customer sites 
e Customer data must be protected (encrypted) 
when carried off-site 


Practises 


All Solaris client computers in our environment 
are identical. Users do not get privileges to install or 
delete software outside their home directories. While 
this may seem unnecessarily restrictive it allows our 
team to scale: identical systems mean identical prob- 
lems and identical solutions; this enables us to make 
sure the desktop is highly available. 
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We decided to apply the same principle for the 
pilot of our new platform. This allowed us to manage 
the software loadsets installed during the pilot. How- 
ever, some software couldn’t be installed on all sys- 
tems so some kind of modular setup looks more feasi- 
ble for a full rollout. 


Mac OS X Laptops 


In order to choose the right platform a list of 
requirements was made and a technical analysis was 
performed. The platform scoring the highest in the anal- 
ysis was piloted. If no show stoppers were found a 
wider beta rollout would be initiated to find any remain- 
ing issues before migrating the entire organization. 


The method used to analyze the various platform 
choices was a weighted-score analysis. Each item in 
the list of requirements is given a weight factor. A 
high weight factor means the requirement is impor- 
tant. Each platform is then given a score for that 
requirement, higher scores mean a better result. 


Fixed desktops scored low because of the lack of 
mobility. Thin clients have higher mobility but aren’t 
truly portable. Windows [4] scored low on manage- 
ability and UNIX compatibility. Linux laptops seemed 
unfeasible due to the lack of support from laptop ven- 
dors.2 Mac OS X scored high for user installable apps, 
hotplugging sound devices and screens and for its 
overall user interface. 


So the results clearly suggested an Apple Power- 
Book platform, mainly because of the fact that it is the 
only viable portable UNIX platform in the list; porta- 
bility and UNIX capabilities had high weights 
attached in the analysis. 


Radmind 


No matter what client platform is chosen, we 
need to manage its disk image. In the case of Apple 
the package management system doesn’t meet our 
needs because: 

¢ Most Mac OS X software is distributed as an 
application inside a compressed disk image. To 
install the software the user simply drags the 
application onto the /Applications folder. There 
is no package information stored in this case. 

e Apple’s package management system has no 
uninstall capability (even if the software is dis- 
tributed as a package). 

¢ There is no tripwire-like functionality to warn 
us about changes to the filesystem. 


Many open source tools exist to create and dis- 
tribute packages. They lack a tripwire [13] functional- 
ity. Radmind [12] was the only toolset that either ful- 
filled the requirements or could be easily altered to 
fulfill the requirements. 

Radmind’s tripwire capabilities [10, 16] trap 
changes to the filesystem caused by package installs 


2Note that Tadpole Computer [6] now offers both Linux 
and Solaris laptops. Sun also sells Solaris laptops. 
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on the Mac OS X machine. This allows us to use any 
method to install a package and record the changes 
made to the system in a transcript. 


FileVault 


All customer data must be encrypted when car- 
ried off-site. There are two ways this can be tackled: 

1. Encrypt the entire hard disk. This makes sure 
no hard disk data is accidentally revealed but it 
also means that once the system has booted, all 
data is available. Note that this causes an 
increase in load times as you’re encrypting the 
entire operating system. 

2. Encrypt user data only. Obviously, there is less 
overhead in this scenario. It can be argued that 
such a modular approach is inherently more 
secure: as soon as the user logs out their data 
becomes unavailable. 


No vendor-supported out-of-the-box solution 
was found for Linux. Apple’s FileVault feature uses an 
AES-128 encrypted autosizing disk image to store the 
user’s homedir. While it wasn’t designed with double 
digit gigabyte sizes in mind it scales reasonably well 
in our experience. The only challenges we had were: 

e There is no out-of-the-box way to enforce its 
use. We ended up scripting the forced creation 
and maintenance at login time. 

¢ Passwords mysteriously change. This is easily 
fixed, provided you’ve installed a ‘master 
password’’s that can unlock any filevault on the 
computer. 

e The disk image needs to be compacted every 
once in a while, which takes a Jong time and 
can only be done at logout. 

All in all we are quite happy with this solution. 


Asset Tracking 


There are good reasons to track a system and its 
users: 
¢ the computer needs to be returned to the lease 
company when the lease ends 
¢ knowing who to ask for the system when you 
need it back 
e if the machine was used in some illegal act it 
helps if you know where it was and who was 
using it 
¢ your organization requires recording of all user 
logins 
The at the end of this section shows the attributes we 
wanted to know about. 


Somehow we’ll need to get mechanisms in place 
that give us the information we need. 


At the time we couldn’t find any off the shelf 
solution to manage our systems. We implemented our 
own software, AssetInterface. AssetInterface is a net- 
work client that stores asset information in a central 
database. The information is stored locally until the 


3This can be configured in the Security preference pane 
in Mac OS X’s System Preferences. 
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database server becomes available — i.e., when the 
client is on the intranet. 


The information in the asset database is also use- 
ful for support; it provides us with a history of DHCP 
addresses for the machine. We can login to a user’s 
machine without having to ask for the IP address. 


The person normally using this 
machine and responsible for 
returning it 


The person actually using this 
machine 


location Where is the machine? 

system image | What is the version of the OS? Are 

release all updates installed? 

serial number | The hardware serial number of the 
computer 

ip address Current IP addresses of the 
computer (wireless, ethernet, .. .) 

mac address | Current MAC addresses of the 
computer (wireless, ethernet, .. .) 


Directory Services 






Attribute 


primary user 
















login user 





Mac OS X can be configured to use a wide range 
of directory services such as NIS [7], LDAP [15] and 
Active Directory [8]. Since we had an existing LDAP 
infrastructure for our Solaris systems we used it for 
our Powerbooks too. 


One noteworthy point is that we duplicated the 
LDAP tree layout of Apple’s Open Directory http:// 
www.apple.com/server/macosx/features/opendirectory. 
html product and pushed it into our existing LDAP 
server as a separate subtree. The regular tree contains 
the identification and authentication data. The Open 
Directory subtree contains authorization data encoded 
as MCX records. 


This setup allowed us to use Apple’s Workgroup 
Manager application to configure the MCX features, 
discussed further. This requires some schema changes 
on your LDAP server.* You can bind to any unmodi- 
fied third party LDAP server but this leaves you with- 
out the MCX feature set. You can also setup your own 
OpenDirectory server just for MCX management, 
leaving authentication to, e.g., Active Directory. 


Managing the System State 


There’s more to managing a system than just 
controlling the contents of the harddisk. Strictly 
speaking, the system state consists of: 

¢ the system hardware 
¢ the state of all bits in memory and on the hard- 
disk 
e the CPU and hardware state 
4LDAP administrators don’t like making schema changes — 
make sure you ask nicely. 
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Luckily, we can control most aspects of the system 
though its harddisk image. For example, 
® scripts run at boot time can change the PROM 
password or update firmware 
e pre-apply scripts can unload kernel extensions 
before upgrading a kernel driver 


File System Management 


Radmind was conceived to help system adminis- 
trators manage lab systems, it’s typically run at logout 
or system boot. Scripts chain together the radmind 
tools to download loadsets, check for differences, 
apply changes, etc. 


It works well on laptops because all network 
connections originate from the client. 


In what follows we outline some of the changes 
we had to make to suit radmind to our needs. 
Scripting 

The clients run a script from cron to check for 
updates and install them. We divert from common 
practice in several ways: 

e We run this script while the user is active — this 
allows updates to happen over VPN links. Con- 
trary to popular belief, this approach caused 
very few problems.° 

© Once updates are ready to install the user is 
notified of the pending update. They can then 
choose whether to allow the update to take 
place at that time. The notification to the user 
includes details such as the expected download 
size and whether the update requires a reboot. 
This allows the user to make an informed deci- 
sion — which turned out to be very important: 

1. users need to feel empowered 

2. there may not be enough time for the 

update to install 

3. the user might be in the middle of some- 

thing important and doesn’t want to be dis- 
turbed 

e Pre-apply and post-apply scripts are down- 
loaded before any other loadsets and before the 
pre-apply scripts are run. 

¢ When an update requiring a pre-apply script is 
pushed out, the pre-apply script can be bundled 
with the update. This is not the case with rad- 
mind which requires a three step scenario to 
accomplish the same task: 

1. push out the pre-apply script 

2. wait for all machines to receive the update 

3. push out the update that needs the pre- 

apply script 

Laptops can be out of touch for weeks on end 

so it’s not practical to wait for step 2 to com- 

plete. 

¢ The example script provided with radmind 
always runs a file system check. Ours only does 


5Note that on Mac OS X, Software Update also installs 


software while the user is logged in. 


19th Large Installation System Administration Conference (LISA ’05) 85 


Herding Cats: Managing a Mobile UNIX Platform 


so when there are new loadsets downloaded from 
the server — keeping clients’ system load low. 
¢ The process checking the file system is run at 
low priority to reduce user impact. 
© We automatically create unique SSL certificates 
for each client machine. The server uses the 
common name (CN) inside the client’s SSL cer- 
tificate to decide which command file to 
present to the client. We can use this later on to 
setup test environments or to specify different 
system images for different platforms.§ This 
also makes it easy to read and grep through sys- 
log files (many SSL services log the CN of the 
client certificate). Automating the creation and 
installation of the unique SSL key is included 
in the asset tracking solution. 
Each application has its own transcript (and 
negative) so that: 

o a modular approach can be introduced later 
on (allowing the user to select which 
optional transcripts to install) 

o single applications can be easily added or 
removed without touching existing tran- 
scripts 

Config files for applications and the OS are 
kept in separate transcripts so that we don’t lose 
the config files when we upgrade to a new 
release of the transcripted software. 


Prebinding 


Apple chose prebinding as a way to improve applica- 
tion launch times.” 


On Mac OS X a typical binary uses functions 
from 10 or more different libraries or “frameworks,” 
each of which typically binds to many more frame- 
works. The net effect is a slow system because of all 
the run-time dynamic binding that has to be done by the 
dynamic loader each time an application is launched. 


One way to get around this is to do most of this 
work at installation time rather than at run time. The 
bindings are performed and the result is saved inside the 
binary. The next time the binary runs the bindings only 
have to be redone if the target of the bind has changed. 


This clashes with a radmind-controlled installation 
because of the install-time prebinding: when you install 
an OS update that updates the C library, 80% of the exe- 
cutables change with it. When you upload these changes 
as a new transcript you end up overruling a large por- 
tion of the base loadset — though all that changed was 
the prebinding information inside those binaries. 


We had to change Radmind so that it would cal- 
culate a file checksum that remains constant through- 
out prebinding changes. The patch is partly based on 


The radmind config file allows wildcards for the host- 
name comparison. We chose a common name that looks 
like “ppc7450.PowerBook15.W654398KQZ4.johndoe” . 
For example, we could give a special image to *.johndoe 
or ppce7450.*. 

TSeveral Linux distributions do something similar. In the 
Linux world, it’s called “‘prelinking.” 


Thibaut and Mertens 


ctool® and is pending inclusion in the standard rad- 
mind distribution. 


Apple will eventually move away from the cur- 
rent prebind method: in the future, prebind informa- 
tion will be stored in a separate directory maintained 
by the OS. 


NetInfo 


One thing you can’t manage directly through 
radmind is NetInfo. It’s a database that contains over- 
riding values for all name services going from 
accounts to automatic mounts. It has the highest prece- 
dence and cannot be disabled. We ended up making a 
post-apply script that keeps the NetInfo database in 
syne with a template. This way we can use radmind to 
make changes to the system accounts and groups. 


User Configuration Management 
User Preferences 


Most Mac OS X applications use the standard 
preferences storage system.? 


This is a very clean system that allows the sysad- 
min to setup default values for any preference a pro- 
gram uses, using a series of overriding locations and a 
standard preference file layout. For example: an applica- 
tion called Bar developed by the Foo company stores its 
preferences in ~/Library/Preferences/com.foo.bar.plist . 
You can provide default settings by storing a file of the 
same name in /Network/Library/Preferences, /System/ 
Library/Preferences or /Library/Preferences (the latter 
override the former). 


If you want to override settings instead of just 
provide defaults you'll need to use the MCX infra- 
structure discussed below. 


Unfortunately, not all applications use this pref- 
erence system. These applications need to be managed 
in a more ad-hoc manner, if at all. 


MCX 


MCX (Managed Clients for OS X) allows con- 
trol over many settings on OS X. For instance, you 
can specify which preference panes a user is allowed 
to open or override any preference for any application 
that uses Mac OS X’s native preference framework. 

¢ MCX also controls credential caching. You 
have to use MCX to make OS X store a local 
copy of a users’ credentials, known as a Mobile 

User account. 

¢ Portable Homedirectories also need to be setup 
with MCX. 


You can find MCX configuration examples on our 
website. More information can be found at Macenter- 
prise.org: http://macenterprise.org/content/view/61/42/. 


8Ctool calculates hashes of binaries. The checksum doesn’t 
change when the prebinding on the binary is redone. See the 
website http://ctool.darwinports.com/ . 

9See Apple’s developer website _ http://developer.ap- 
ple.com/documentation/CoreF oundation/Conceptual/CF- 
Preferences/ for details. 
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Supporting Infrastructure 


Global radmind Deployment 


For the server side of radmind, standard practice 
is to have one server that runs one radmind process on 
one port. 

e In order to scale globally we have one radmind 
server per main site. They each serve files off a 
volume that is mirrored across all sites. 

° The master volume is kept on a separate host 
that doesn’t serve files. This makes sure we can 
make changes to the volume without interfering 
with the radmind server processes. 

© We run three radmind instances per server: the 
stable, testing and staging releases. This allows 
us to test command files and radmind configu- 
rations before we change the stable release. 
Some of radmind’s directories on the server can 
be shared between instances where appropriate. 

© We created scripts for distributing changes from 
the master that make sure that all files are con- 
sistent before sending them to the slave servers. 

e Changes to radmind’s code can alter the layout 
and contents of the transcripts. In such cases we 
push the new radmind to the stable clients. In 
the same update we change the port number of 
the server where the new transcripts can be 
found. When all systems have upgraded in this 
hop-skip way, we upgrade the stable release 
and point the clients there again. 


Backups 


All relevant user data is regularly synchronized 
to the Solaris NFS server using a home grown rsync 
[14] front-end. The server data is periodically backed 
up to tape removing the need for a separate, native 
backup solution on Mac OS X. This also puts the data 
within easy reach on the NFS server where the users 
can get to it from their Solaris desktops. In our OS X 
10.4 release we plan to use the built-in Portable 
Homedirectories instead of the rsync front-end. 
Reporting 

Some errors trigger an email report program 
(surprisingly named “‘mail-report”) that automatically 
mails the admins with the error output. This email 
includes network configuration information allowing 
us to login to the system remotely and fix the problem 
before the user even notices it. We changed the mail 
configuration so it delivers the mail as soon as the user 
connects to the intranet. /etc/postfix/main.cf (postfix’s 
main config file) is auto-generated from a template, 
this way we can fill in the hostname in the config file 
using a script. 


Open Issues 


Radmind Atomicity and Bandwith Use 


Radmind’s lapply phase downloads and installs 
files directly onto the live file system. It would be 
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better if it downloaded all changes first and then 
applied them. This way updates don’t fail somewhere 
in the middle when the user removes the network 
cable. 


Radmind conserves bandwidth by only down- 
loading changed files. The network connection to the 
server is not compressed, however. We are working on 
zlib compression for the data channel. 


“Don’t Care” Option 


Radmind can provide defaults for a file. If the 
file isn’t present on the system it is downloaded. If it is 
present its content is not checked — only its permis- 
sions are. We wanted a more versatile ignore com- 
mand in radmind so some files would be ignored com- 
pletely. A patch is available on our website. 


Software Install 


Some software needs to be installed on the root 
partition. Even if the user is given permissions to 
install applications there, the files will simply be 
removed at the next radmind-update. One possible 
solution is to record such installs using radmind. The 
loadset is added to those downloaded from the server, 
allowing local overrides to the base image. We hope to 
have this solution by the start of the next project phase. 


Configuration Management 


Full configuration management wasn’t a priority 
during the pilot. We'll need to implement such a sys- 
tem later to cope with the many different setups and 
configurations that are possible. There are many 
projects that attempt to address issues like this, e.g., 
LCFGng [9] and CFengine [11]. LCFGng uses RPM 
packages to install and maintain the system. It can 
likely be modified to use radmind transcripts instead, 
adding a powerful security layer that can detect and 
fix issues automatically. 


Lessons Learnt 


° Most package management tools are not written 
with fault tolerance in mind. They assume that 
they know the state of the system without 
checking it. Even if they include a “package 
checks” tool, they don’t allow you to restore 
missing or altered files. 

¢ Tripwire-like tools operate on the filesystem — 
not on an assumed state of the filesystem. 
Because of this they are inherently fault toler- 
ant. Other package tools should consider imple- 
menting a fast way to: 

o verify the filesystem contents versus the 
package descriptions 

o fix inconsistencies and compromised com- 
ponents 

e OS X is a great platform if you don’t want to 
give your users administrator privileges. Many 
tools and frameworks are in place (and stable) 
that give users control over their systems while 
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keeping the reigns firmly in sysadmin hands. 
On top of that, most third party applications can 
be installed without privileges by simply drag- 
ging and dropping them on the user account. In 
our case it was possible to give our users lim- 
ited sudo capabilities only. This didn’t seriously 
limit their ability to do their jobs. 

Radmind gave us the ability to see what horri- 
ble things some third party drivers did to our 
precious image and allowed us to compensate 
accordingly. 

Keeping transcripts clean and modular can be 
difficult but pays off when you upgrade or dis- 
able applications. 

Giving users the ability to select which soft- 
ware they want installed on their systems is 
important for a full roll out. However, a pilot 
project can afford not to bother with that and 
can provide the same system image to all 
clients instead: the needs of the many outweigh 
the needs of the the few — or the one. 


Conclusion 


A mobile Mac OS X platform can be adequately 
managed using radmind. Leased laptops require some 
form of asset tracking software — a solution had to be 
written in-house. 
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Open Network Administrator (ONA) — A 
Web-Based Network Management Tool 


Bruce Campbell and Robyn Landers — University of Waterloo 
ABSTRACT 


This paper presents Open Network Administrator (ONA), a web-based network management 
tool. Network administrators interact with ONA over the web, and ONA interacts with routers, 
switches and access points, using telnet and/or SNMP. 


ONA provides a common web interface to a number of different vendors’ network products, 
and provides granular access control, permitting network administrators across multiple 
departments to manage a network in a consistent manner. In addition to typical day-to-day 
operations like changing switch port speed, duplex, VLAN, etc., ONA performs a number of other 
maintenance operations, such as automatically backing up switch configurations, maintaining 
traffic graphs, and sending device reboot/up/down alerts via e-mail. 


ONA leverages some existing components, such as RRDtool [1] and CVSweb [2]. A summary 
of existing tools, both commercial and open source, is provided, with comparisons made to ONA. 


At the University of Waterloo, where there is both a central IT group and many sizeable 
decentralized autonomous IT groups, ONA rapidly gained widespread enthusiastic acceptance. 


Introduction 


The managed network switch offers significant 
advantages, including the ability to control individual 
port settings, speed/duplex, VLAN, etc. But there is a 
caveat: managed switches need to be managed. For a 
small network, the web-based management tool 
included with most switches is adequate. But as the 
network expands, as IT staffing grows, and as the 
number of features packed into edge devices 
increases, a need to treat the network edge as a system 
develops. Managing each switch independently 
becomes unwieldy. 


The movement of network features from the core 
to the edge has not been matched by a corresponding 
increase in availability of edge device management 
tools. The traditional approach to edge device man- 
agement is to try to keep the edge simple, and do 
everything smart in the core. This leaves powerful fea- 
tures of modern switches unleveraged, and ultimately 
leads to scalability issues. 

Abrahamson, et al. [15] point out the importance 
of integrating a definitive source of edge device con- 
figuration data into a software tool that is easy to use 
and of significant benefit to its users. Many network 
vendors do offer a comprehensive network manage- 
ment tool which can do just about everything imagin- 
able... with that vendor’s latest and greatest network 
equipment. The prospect of upgrading an enterprise’s 
entire network infrastructure, and then being tied to a 
single vendor, often makes this option unrealistic. 


ONA can be dropped into an organization’s 
existing heterogeneous network, with existing equip- 
ment, and existing network management structure (or 
lack thereof), and provide an immediate benefit. 


Motivation 


In 2000, the networking for the Davis Centre 
building at the University of Waterloo was upgraded 
from thinnet ethernet to switched ethernet. The net- 
working in this building was managed by the IT group 
for the Faculty of Mathematics. 


Extreme Networks hardware was selected, and it 
was initially managed through the Extreme Networks 
management tool, EPICenter [3]. 


The building also hosted researchers and admin- 
istrative staff from the Faculty of Engineering, which 
had its own IT staff. In order to streamline the day-to- 
day network management activities (i.e., moves, adds, 
and changes), a number of IT staff from both faculties 
involved were given administrative access to EPICen- 
ter. Procedures were developed for documentation and 
change notification. 


EPICenter did not support the notion of shared 
management well, and logging was minimal. There 
was no way to restrict the types of changes each staff 
member could make, and tracing back changes to a 
specific person was difficult. The learning curve for 
EPICenter was fairly steep, and the user interface 
made it easy to make errors. This created a need to 
give EPICenter administrative access to as few IT staff 
as possible. 

Conversely, with a major network upgrade in 
progress, the number of moves, adds, and changes was 
significant. This created a need to give EPICenter 
administrative access to as many IT staff as possible. 


We needed a tool that permitted IT staff to make 
the changes they needed to make, and prevented them 
from making changes they shouldn’t make. 
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Requirements 


The minimal requirements were for a secure, 
easily usable and accessible tool to permit network 
support staff to change speed, duplex, VLAN, etc., 
settings on network switch ports for which they have 
been authorized, and to log all such changes. 


The requirements for being secure and easily 
accessible made web-based operation the preferred 
choice. Although it is possible to install custom soft- 
ware on the desktops and laptops of all network sup- 
port staff, or to use X11 or RDP to connect to a server- 
based GUI application, these are considered signifi- 
cantly less practical than web-based operation. 


Prior to investigating a switch port management 
tool, we had already developed systems to perform a 
number of other network related tasks (e.g., maintain- 
ing traffic graphs with MRTG [4], sending device 
down/reboot alerts with Nagios [5]), as well as some 
custom scripts for saving switch configurations to a 
TFTP server. Each of these systems had its own list of 
switch names and SNMP community strings. This was 
not ideal, and led to the requirement that the new tool 
also perform these tasks, such that our existing sys- 
tems could be phased out. 


Finally, it was recognized that networking tech- 
nology and the way we use and manage it is con- 
stantly evolving. This led to a requirement that the 
new tool be extensible to meet currently unforeseen 
needs. Therefore, while not strictly a requirement, an 
open source solution would be viewed as preferable to 
a commercial solution. 


Alternatives 


As of the day of writing, there is no web-based, 
open source, network management tool to be found. 
Most of the network management tools available are 
not in fact device management tools. The areas of DNS 
management, network monitoring, and traffic graphing 
appear to be well covered. They break out like this: 

¢ LANdb, a network documentation tool, appears 
feature rich according to its web site, and even 
promises device management in a future release. 


PURPOSE 
traffic graphing 
DNS manager 
switch manager 


cacti [6] 

IP Manager [7] 
Kiwi CatTools [8] 
LANdb [9] 
WhatsUp gold [10] 
network manager 
network manager 
network monitoring 
switch configuration 
switch configuration 
network monitoring 


HP OpenView [11] 
GxSNMP [12] 
OpenNMS [13] 
RANCID [14] 
splat [15] 

SNMPc [16] 


network documentation 
network monitoring, mapping 
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However, development appears to have been 
stalled since 2000. 

GxSNMP is a device management tool, albeit 
not web-based. Again, development appears to 
have been stalled since 2001. 

Kiwi CatTools is a popular commercial Win- 
dows-based tool for managing, monitoring and 
reporting on network devices and their configu- 
ration. A limited version can be downloaded for 
free from their website, with more powerful 
versions aggressively priced. 

HP OpenView is perhaps the most widely known 
and powerful enterprise network management 
tool. Pricing is targetted towards the enterprise. 
Splat is a network device management tool with 
capabilities generally similar to ONA. The most 
obvious difference is that Splat is command-line 
oriented while ONA is web-based. ONA satisi- 
fies most of the requirements listed by the 
authors of Splat [15]. Splat uses RANCID [14] 
for secure communication with devices; ONA 
uses authenticated SNMP/telnet. Splat relies on 
an authoritative inventory database to obtain 
MAC addresses and hostnames. This is not prac- 
tical for ONA because ONA is used by many 
departments in several faculties at UW, each of 
which might have its own inventory database 
design, or none at all. ONA implements features 
such as multi-level access control, automated 
switch recovery, and initial switch configuration, 
which were future goals for Splat. 


Design 


ONA is a web-based application, written in PHP 
{17]. It interacts with network switches, routers, and 
access points, using SNMP and/or telnet scripting. 
Device configuration and data are stored in a MySQL 
[18] database. 


The database consists of approximately 40 tables 
that store definitions (administrators and devices), 
group memberships, state information (ARP and MAC 
tables), and logs. The devices table contains the 
device host names, device types (switch, router, wire- 
less access point, etc.), primary group membership, 


INTERFACE AVAILABILITY 


web open source 
web open source 
Windows GUI commercial 
web open source 
web, GUI commercial 
web commercial 
Linux GUI open source 
web open source 
script open source 
script open source 
web commercial 


Table 1: Existing management tools. 
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and SNMP community strings. The administrators 
table contains the userids, primary group membership, 
and preferences for each administrator. The sections 
on Granular Access Control, Change Logging, and 
MACHP. Querying, Logging, Searching explain the 
group membership tables, state information tables, and 
logs, respectively. 

ONA uses Apache [19] .htaccess for user authen- 
tication, and thus any underlying authentication mech- 
anism (e.g., RADIUS [20], LDAP [21]) can be used. 
ONA receives the successfully authenticated userid 
from Apache, and does not interact directly with 
authentication servers itself. 


While VLAN and port settings from all devices 
are stored in the ONA MySQL database, the configu- 
rations on the switches themselves are still considered 
authoritative. That is, if a switch configuration is 
changed manually, ONA will not overwrite that 
change, but rather will update its own database to 
reflect the change, either within 24 hours, or immedi- 
ately ifan ONA administrator chooses that option. 


An alternative design which would make the 
ONA database authoritative was also considered. This 
could provide some advantages, such as the ability to 
make global configuration changes and have them 
automatically pushed out, or to replace a failed switch 
and have its configuration automatically updated. 
While the technical advantages were tempting, it 
would increase the risk that a bug in ONA could push 
a damaged configuration out to a large number of 
devices. Our experience with a commercial wireless 
access point manager has demonstrated this risk. We 
manually upgraded the firmware on a number of 
access points to fix a bug that was affecting our wire- 
less deployment. The management software did not 
support the newer firmware, and rather than reverting 
to a “do not understand — do not do anything” state, it 
pushed an unusable configuration to the upgraded 
APs, putting them offline. This type of experience has 
the potential to chill enthusiasm, and was considered 
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too risky for an environment that depends on volun- 
tary adoption of solutions. 


ONA is not a network documentation tool in 
itself. In addition to the standard port description field, 
ONA supports a comment field, which is stored in the 
ONA database. This field can be used to hold a jack 
number, hostname or other information as desired. 


A word about terminology. An ONA administra- 
tor is someone who uses ONA to do network adminis- 
tration. An ONA manager is someone who has author- 
ity to manage or configure ONA itself (including for 
example granting permissions to administrators). 


Main Features 


Multi Vendor Support 


ONA currently supports Extreme, HP Procurve, 
Nortel Baystack, Cisco 2900x1/3500x1 and Cisco 
2950/3550 switches, and Avaya AP-3 access points. 
Adding support for a new device involves writing 
PHP code to perform a number of primitive operations 
on the device, like fetching the port speeds and 
duplexes, setting tagged VLANs, etc. For example, the 
functions for managing the Nortel Baystack switch are 
as shown in Display 1. 


The bulk of the work involved in adding support 
for additional devices is investigation of the SNMP 
involved. Beyond basic functions like querying the 
port statistics and interface descriptions, different ven- 
dors’ SNMP implementations have little in common. 


Each vendor typically relies on its own private 
MIBs, even for generic settings such as port speed and 
duplex. This is unfortunate as it reduces the opportu- 
nity to leverage existing code to support multiple ven- 
dors’ equipment. 


To give an idea of the effort required to add sup- 
port for new devices, a student investigated the SNMP 
involved for the HP Procurve line over a period of two 
weeks, and then one of the authors developed the 
code, from scratch to debugged and in production, in 
about five days. 


function set_nortel_port_tagged_vlans_via_snmp( $d, $portname, \ 
Solduntaggedvlan, Suntaggedvlan, Soldtaggedvlans, Staggedvlans ) 
function set_nortel_port_untagged_vlan_via_snmp( $d, S$portname, \ 


Soldvlan, $vlan, S$istrunk ) 


function adjust_nortel_vlan_members( $d, S$vlan, S$remove_this_port, \ 


Sadd_this_port ) 


function set_nortel_port_trunkmode_via_snmp( $d, Sportname, \ 
Strunkmode, Solduntaggedvlan, Suntaggedvlan, Soldtaggedvlans, \ 


Staggedvlans ) 


function get_nortel_vlan_configuration_via_snmp( $d, Ssignature ) 
function get_nortel_port_speeds_and_duplexes_via_snmp( $d, Ssignature ) 
function set_nortel_port_speed_duplex_via_snmp( $d, S$portname, \ 


Sspeed, Sduplex ) 


function get_nortel_model_and_version_via_snmp( &$d ) 


function nortel_telnet_login( Sd, Scontin ) 


function nortel_telnet_logout () 


function create_nortel_vlan_if_needed( $d, 


Svlan ) 


Display 1: Functions to manage the Nortel Baystack switch. 
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Traffic Graphs with RRDtool 


RRDtool is integrated with ONA, and ONA 
automatically maintains RRD databases for all switch 
ports, via a cron job that runs every five minutes. Traf- 
fic graphs (or traphs as we like to call them) can be 
displayed in several formats. A summary traph page 
for each switch shows a 24-hour graph for each port. 
For each port, a number of traphs are available, rang- 
ing from a period of six hours to one year. A real time 
graph tool can show traffic on a specific port, updated 
every ten seconds. The real time traphs also show 
packet counts and error counts. An example summary 
traph page is shown in Figure 1. 

Granular Access Control 


Permissions are managed by means of relation- 
ships between administrators and devices. Administra- 
tors and devices are listed in tables that associate them 
with various groups. The groups are typically organi- 
zational units such as departments, faculties, IT sup- 
port groups, and so on. 


Devices and administrators each have a primary 
group to which they belong. The devices and admin- 
istrators tables establish these primary group mem- 
berships. Additional group memberships are assigned 
through supplementary tables. Regular expressions 
may be used in these tables to specify, for example, 
ranges of ports, or devices whose names start with a 
common prefix, and so on. Each device (switch, 
router, or access point), switch port, VLAN tag, and 
administrator must be a member of at least one group. 


4 ONA - Microsoft Internet Explorer, 
Eile Edit View Favorites Tools Help 


<p Back ° ty ¢ ' yp Search S ” Favorites & 
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Permission for an administrator to take a certain 
action on a certain device is determined by analyzing 
the group membership relationships. For example, an 
administrator may change the speed or duplex on a 
switch port if they have a group in common with the 
port or switch. They may change an untagged VLAN 
on a port if they have a group in common with the 
VLAN to which the port is being changed. If the port is 
a trunk port, then they also must have a group in com- 
mon with all tagged and untagged VLANs on the port. 


Every button in ONA and every settable parame- 
ter may be individually disabled or enabled on a per 
administrator basis. As an example of the level of 
granularity that this access control mechanism pro- 
vides, an administrator could be given permission to 
change speed and duplex only, on one port only, and 
not have access to any of ONA’s other features. 


The following example illustrates how ONA 
uses groups and regular expression pattern matching. 

¢ Two departments, “Engineering” and ‘“‘Sci- 
ence,” each have 10 switches. 

e A core device “core-swl” is managed by a de- 
partment called “‘Central Services Department” 
(CSD) 

¢ The Engineering switches are named eng-sw1 
through eng-sw10 

e The Science switches are named sci-sw1 through 
sci-sw10 

e vlans 10 and 11 carry Engineering networks 

e vlans 20 and 21 carry Science networks 
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Figure 1: Summary traffic graph page. 


92 19th Large Installation System Administration Conference (LISA °05) 


Campbell & Landers 


> 


© The userid ‘‘engadmin,” who works in Engi- 
neering, has full access on all Engineering 
switches 

© The userid “sciadmin,” who works in Science, has 
full access on all Science switches 

e A user “manager,” who works in CSD, has full 
access on all switches 

¢ A user “operator,” who works in CSD, has per- 
mission to disable/enable client ports on all 
switches 


Implement the above using the managerial inter- 
face, as follows: 

e Under “Devices,” add eng-swl through eng- 
sw10, setting the groupid to “Engineering” 

e Under “Devices,” add sci-swl through sci- 
sw10, setting the groupid to “Science” 

e Under “Devices,” add core-swl, setting the 
groupid to “CSD” 

e Under “‘Administrators,”’ add ““engadmin,” set- 
ting the groupid to “Engineering” 

¢ Under “Administrators,” add “sciadmin,” set- 
ting the groupid to “Science” 

¢ Under “Administrators,” add “manager,” set- 
ting the groupid to “CSD” 

e Under “Administrators,” add “‘operator,” set- 
ting the groupid to “CSD,” setting denytrunk- 
changes to “1,” and setting allowededits to 
“portstate”’ 

e Under “Device Group Memberships,” add a 
groupid called ‘‘all,”’ setting the ipname to *.” 
(the regular expression consisting of a single 
period matches everything) 

ZONA - Microsoft Intermet Explorer. 
Eile Edit View Favorites Tools Help 


Deak - x} 2) @D JO seach 


Favorites 4) 
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© Under ‘“Vlan Group Memberships,” add a 
groupid called “all,” setting the vlan to “.” 
(the regular expression consisting of a single 
period matches everything) 

e Under “Administrator Group Memberships,” 
add two entries, one for each of “manager” 
and “operator,” setting the groupid to “all” 

¢ Under “Vlan Group Memberships,” add a 
groupid called “Engineering,” setting the vlan 
to *(10|11)$ (that is a regular expression which 
matches 10 or 11) 

¢ Under ‘“‘Vlan Group Memberships,” add a 
groupid called “Science,” setting the vlan to 
*(20|21)$ (that is a regular expression which 
matches 20 or 21) 


The above can be taken further. For example, 
engadmin and sciadmin could be given control over 
their respective ports on core-swl, through use of 
“Port Group Memberships.” Or, if Engineering and 
Science wanted to share switches in a given area, there 
are several ways to use group memberships to permit 
such sharing, without granting full access to non- 
shared switches. 

Change Logging 

For each change, the date, time, userid, IP 
address of the client, the parameter being changed, its 
previous value, and its new value, are all logged. For 
each device, a “Changes” button lists all changes in 
chronological order (See Figure 2). On the port edit 
screen, the changes for that particular port are dis- 
played at the bottom of the screen. (All history is kept 
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Figure 2: History of device changes. 
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indefinitely. The need has not yet arisen to truncate 
history or roll older portions behind a “‘more”’ button.) 


Managerial changes (e.g., adding a new switch, 
changing an SNMP community string), and batch tel- 
net commands, are also logged, but these logs are not 
available for display through ONA at this time. 


All ONA logs are stored in SQL tables. The 
structures of the adminlog, portchangelog and telnet- 
log are shown in Tables 2, 3, and 4. 


These features make use of the above tables: 

¢ The switch display screen shows MACs and 
IPs on each port. For trunk ports, the number of 
MACs seen in the last seven days, and a link to 
the complete list are shown (see Figure 3). 

The MAC/IP search tool shows all ports the 
MAC has been seen on (including trunks), with 
a quick link to the port if the MAC was found 
uniquely on that port. Also shown is the MAC, 
ARP, and port history. (e.g., if a network card 
was changed in a computer, ONA will show 
when that happened, both in terms of the 
change to the ARP table, and the MAC address 
present on the port); see Figures 4 and 5. 


MAC/IP Querying, Logging, Searching 
A cron job queries the MAC tables (and ARP 


tables for routers) regularly throughout the day. The data 
are stored in several tables, as shown in Table 5. 


Switch Configurations and CVS Repository 


A nightly cron job saves the switch configura- 
tions to a TFTP server, and can optionally push the 
configurations to additional TFTP/FTP servers. The 
configurations can also be placed in a tar ball, for auto- 
mated pulling to network operations staffs’ laptop com- 
puters. (Instructions for doing this with Cygwin [22] are 
included in the documentation for ONA.) 


The pushing of switch configurations to alternate 
servers can be done based on the primary groups of the 
devices. For example, the configurations for switches in 
group “Engineering” could be pushed to a specific 
server belonging to Engineering. 

Plain text device configurations are also saved to 
two different CVS repositories. One contains the raw 
configurations, and is not made available through ONA 
or CVSweb. The other contains the configurations 
minus sensitive information, such as community 
strings. The latter is available through ONA, with a link 
to CVSweb. 

Managerial Interface 

ONA managers (administrators with the “systemad- 
min” setting) can add/change/delete administrators and 
devices, and can adjust group memberships, etc., through 
the managerial interface, as shown in Figures 6 and 7. 
Up/Down/Reboot Alerts 


ONA logs and e-mails alerts when devices reboot, 
become inaccessible, or reappear. Administrators can 
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disable receiving these e-mail alerts in the ONA prefer- 
ences window. The set of devices about which a user 
receives notification is controlled by regular expression 
manipulation of the groups to which the devices belong. 








DESCRIPTION 


sequence sequence number 
userid userid of person who made the 

















change 
clientip IP address of client used by userid 
info raw SQL (minus sensitive infor- 


mation) showing what was 
changed in the ONA admin tables 
date/time of change 


Table 2: Fields of adminlog. 


FIELD DESCRIPTION 


sequence sequence number 
userid userid of person who made the 


timestamp 













change 
clientip IP address of client used by userid 
device name of device changed 
portname name of port changed 
setting settings, i.e., speed, vlan, etc. 
oldvalue old value 
value new value 


timestamp date/time of change 





Table 3: Fields of portchangelog. 


FIELD DESCRIPTION 


sequence sequence number 

userid userid of person who made the 
change 

clientip IP address of client used by userid 

device name of device changed 

announce Boolean indicating whether this 

change should be announced in 

daily e-mail port (checkbox on tel- 

net interface) 

the telnet command that was run 

date/time of change 


command 
timestamp 





Table 4: Fields of telnetlog. 


arp IP to MAC 
rarp MAC to IP 
arplog changes in the ARP table 
MAC tables, with no analysis, i.e., raw 
data 
mac MAC tables, with results from trunk 
ports removed 
maclog changes in the MAC table 


Table 5: Table names. 


rawmac 
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Figure 3: Switch page. 
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Figure 4: Search results. 
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4 ONA - Microsoft Internet Explorer 
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Figure 6: Managerial interface — main screen. 
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Daily E-mail Report 

Each night, ONA mails a summary of changes, 
alerts, and configuration differences from the previous 
day (minus sensitive information), to the administra- 
tors. Appendix | shows an example e-mail report. 


Batch Telnet Interface 


When specifically authorized in the authgroups 
table, administrators can perform batch telnet commands 
on a device or group of devices, for devices which sup- 
port a command line interface. ONA is programmed to 
understand the telnet interfaces on the devices that it 
supports, so that it can supply the userid and password, 
transparently, and wait for command line prompts 
before sending the next in a series of telnet commands. 


All commands run through the ONA batch telnet 
interface are logged. The administrator can choose 
whether to have telnet commands announced in the daily 
e-mail report, by use of a checkbox labelled Announce. 


Commands run on a switch through telnet have 
the potential to make ONA’s switch configuration 
database out of date. For example, if an administrator 
were to run commands via telnet to change the 
untagged VLAN on a port, ONA would continue to 
show the old VLAN for 24 hours, based on the values 
stored in its internal database. For this reason, the 
ONA telnet interface includes a radio button that can 
force ONA to synchronize its database immediately 


Eile Edit View Favorites Tools Help 
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(Sync Now), or upon the next access to the switch 
through ONA’s web interface (Sync Later). 


Commands entered through the batch telnet 
interface bypass ONA’s granular access control, so 
this interface is disabled by default, and is made avail- 
able only to administrators who already know the 
actual switch passwords. An example of the use of this 
interface would be upgrading the firmware on a large 
number of devices, followed by rebooting the devices. 
The following commands were used to upgrade the 
firmware on approximately 100 Avaya access points: 

download 129.97.50.121 \ 
tmpdata/AVAP3.bin img 
reboot 2 


After entering the above commands into the 
command window, and selecting the 100 access points 
via checkboxes, ONA ran the command on all 100 
access points with no further input required. 


Configuration Translator 


ONA can generate a suggested configuration 
fragment for a device using the text configuration 
commands of a different vendor. A manager may use 
this to relatively easily replace a switch with one from 
a different vendor. 


The configuration fragment generated by ONA 
consists only of the VLAN database, and speed, 
duplex, description, and VLAN(s) of all ports. 


~~. B- 
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Figure 7: Managerial interface — editing administrators. 
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Display 2 shows sample configuration fragments 
for port 1, which is set 100/full, on VLAN 49. 


The ONA Configuration Translator does not 
actually translate a given switch configuration text file 
from one vendor format to another. Rather, it gener- 
ates the configuration fragment based on the VLAN 
and port information stored in the ONA database. For 
this reason, adding support for a new vendor involves 
writing fairly simple code to generate the text com- 
mands specific to that vendor, without knowledge of 
any other vendors’ configuration file formats. 


Security 


Access to the ONA application is controlled 
through Apache .htaccess authentication. A configura- 
tion file contains the MySQL userid and password so 
that the ONA application may connect to the database. 


The database contains the userids, passwords, 
and SNMP community strings for all devices. These 
are stored encrypted in the database, with a key con- 
tained in the configuration file. 


ONA should be hosted on a server that does not 
permit general user login, or host web pages for gen- 
eral users, to help protect against unauthorized access 
to the configuration file and the database. 


Access controls to limit administrative access 
should be applied on switches and access points, 
independently of whether one uses ONA. If a 
switch is stolen, the password or SNMP communi- 
ties may be compromised. If all switches use the 
same passwords and communities, access controls 
restricting access to specific authorized hosts or 
subnets can reduce the impact. 


Having a single network management tool han- 
dle a number of tasks means that userids, passwords 
and communities need be stored in only one location. 
This reduces the administrative burden of entering this 
information into several tools. By reducing this admin- 
istrative burden, ONA makes it more likely that weak 


or widely known community strings will actually be 
changed, thus eliminating the common / was meaning 
to get to that explanation after a security compromise. 


ONA’s access controls limit what functions 
administrators can perform. Even in the case where 
administrators are given permission to do more than 
they actually need to do, ONA’s logs indicate who did 
what and when. 


ONA is also a handy tool when dealing with secu- 
rity incidents. When a computer is suspected of having 
a virus or is otherwise behaving badly on the network, 
ONA’s “search” facility lets an administrator quickly 
search for the machine’s connection point by either 
hostname, IP address, or MAC address. ONA shows the 
switch port on which the machine is connected, and one 
click takes the administrator to that port where it may 
be quickly disabled. A dialog box is available for the 
administrator to enter an explanatory message which is 
then e-mailed to the machine’s user/owner/administra- 
tor as obtained by ONA from DNS. 


Use of ONA At the University of Waterloo 


Network management at UW is decentralized. 
The central IT group, Information Systems and Tech- 
nology (IST), has responsibility for the external cam- 
pus connection, internal campus backbone networking 
infrastructure out to points of presence in the faculties, 
and for all of student residence networking. The facul- 
ties may have responsibility for their own network in- 
frastructure between their IST-provided point of pres- 
ence and the end-user. Faculties such as Engineering 
and Mathematics have fairly large network infrastruc- 
tures of their own; other faculties may have less or 
none (in which case IST provides it). 


Before the development of ONA, network 
administrators used the vendor CLI tools for switch 
management. This meant learning the details of the 
tools for Cisco, Bay Networks (Nortel) and Extreme 
switches among others. Bay offered an ASCII GUI. 
Extreme offered both CLI and the EPICenter tool. 





® Cisco: ¢ Extreme: 
interface FastEthernet0/1 configure port 1 auto off speed 100 duplex full 
speed 100 create vlan "cstclnet" 
duplex full configure vlan "cstclinet" tag 49 
switchport mode access configure vlan "cstclnet" add port 1 untagged 
switchport access vlan 49 © Nortel Baystack: 
—— interface FastEthernetl 
¢ HP Procurve: speed 100 
interface 1 duplex full 
speed-duplex 100-full exit 
exit vlan ports 1 filter-tagged-frame enable 
vlan 49 vlan ports 1 filter-untagged-frame disable 
name "cstclnet" vlan ports 1 tagging disable 
untagged 1 vlan create 49 name cstclnet type port 
exit vlan members add 49 1 
vlan ports 1 pvid 49 


Display 2: Sample configuration fragments. 
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EPICenter was used for initial high level design but 
was abandoned due to its complexity and other short- 
comings described earlier. Permission to modify 
switch settings was restricted to staff members who 
were directly responsible for network infrastructure. 


ONA was well-received by senior network 
administrators. It offered a single easy interface to all 
the supported switches covering most of the device 
functionality. Significantly, it freed up senior network 
staff from the high volume of simple port configura- 
tion changes by enabling other technical staff to per- 
form those simple changes themselves. Vendor CLI 
tools are still used in some circumstances when certain 
functionality is either not available via SNMP/telnet or 
not yet implemented in ONA. 


ONA was developed in one department of the 
Engineering faculty, and was adopted voluntarily by 
most other Engineering departments, the Mathematics 
faculty, and others until at present four of the six facul- 
ties use it. In a decentralized environment where there 
are often tendencies towards preserving local control 
and resisting external influence, this widespread 
enthusiastic adoption of ONA is quite remarkable. The 
central IST department is now adopting ONA and 
plans are underway to bring all remaining campus net- 
work switches into ONA by the end of the year. 


From the period May 1, 2004 to May 1, 2005, 
approximately 10,000 network port changes were 
made through ONA. ONA is the primary mechanism 
for switch port management, to the extent that the 
built-in web-based management tool that comes with 
the switches has been disabled, and is no longer 
depended on. 


Even most veteran network operations staff, who 
are adept at the command line interfaces for multiple 
models of switches, choose to use ONA over telnet- 
ting into the switch directly. 


Maintenance and Support for ONA 


ONA was developed by one person, without the 
expectation in advance that it would be offered as an 
open-source tool to the community. It is implemented 
in roughly thirty PHP source files [23] and admittedly 
needs some polish to make it more readily amenable to 
both collaboration by multiple maintainers, and to out- 
side use. However, it has been successfully installed 
by someone else without a great deal of difficulty. 


ONA depends on having Apache web server, 
MySQL database, ModPHP (with SNMP, MySQL and 
FTP support enabled), CVSweb, and RRDTool 
installed. Then to bootstrap ONA, currently one must 
manually create the first manager and the default set 
of tables in SQL. (These bootstrapping steps are docu- 
mented in detail [24].) 

ONA is actively supported and developed by 
Bruce Campbell with contributions from Dawn Keenan 
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of the Information Systems and Technology group at 
UW. With more recent perspective arising from ONA’s 
enthusiastic acceptance at UW, and with the realization 
that there do not appear to be similar tools evolving 
elsewhere, there has emerged a small group of people 
spanning several UW IT groups who are interested in 
becoming active maintainers of ONA. Also, ONA is 
recognized at the administrative level of the Univer- 
sity’s IT organizations as an important tool worthy of a 
commitment of resources. We expect that these factors 
will lead to ensuring that ONA is maintainable and 
well-supported. Constructive feedback from readers of 
this paper would also help us move towards making 
ONA a well-packaged open-source tool. 


System Requirements 


ONA is written in PHP and has been tested on both 
FreeBSD and Linux. It should work on any UNIX plat- 
form. 


To give some idea of the CPU etc. requirements 
to run an instance of ONA, some details of UW’s pro- 
duction ONA installation are provided. 


The system which runs http://ona.uwaterloo.ca/ 

is as follows: 

e FreeBSD 4.11 

e dual 2.80 GHz Xeon 

¢ 512 MB memory 

e 3ware SATA RAID 
Devices: 

¢ 277 switches (17288 ports) 

e 299 wireless access points 

© 6 routers 


The cron jobs that query the port traffic statistics, 
and query the MAC address tables of all devices, 
place a moderate load on the server, with the load 
average typically sitting around 0.50. 


The cron job that updates the RRD databases 
takes about seven minutes to query and update the port 
statistics for the approximately 17,000 ports. 


The MySQL database is approximately 400 MB, 
the bulk of which is the rawmac table. 


Conclusions 


ONA seems to fill an important niche that is cur- 
rently comparatively vacant, and seems to fill it well. 
It is a secure, easy-to-use, open source, web-based net- 
work switch management tool. 


Its ready acceptance by both traditional command- 
line-oriented “power user” network administrators and 
other IT support staff, in a strongly decentralized set- 
ting, suggests that its advantages are very appealing. Its 
numerous features and the simplicity of its user inter- 
face lead staff to prefer it over traditional methods. 


ONA is designed to be friendly to changes made 


to switches by means other than ONA. This benign co- 
existence permits deployment in a staged manner, in 
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which network operations staff can continue to man- 
age switches directly as desired, and use ONA when 
they decide to. The deployment of ONA at UW did 
not require an overhaul of network management prac- 
tices. ONA doesn’t do everything — some operations, 
such as creating quality of service profiles, access con- 
trol lists, etc., must still be done directly. But since 
ONA is comfortable with device changes made unbe- 
knownst to it, ONA itself does not become an obstacle 
to using device features which it does not yet support. 


ONA reduces the burden of network manage- 
ment by performing a variety of maintenance opera- 
tions and handling the vast majority of typical switch 
port configuration changes. 


It is surprising that there seem to be no other 
open source web-based edge device management tools 
available. We hope others will examine and use ONA, 
give us feedback as we move towards making it a 
well-formed open source tool, or perhaps be inspired 
to develop and contribute their own. 


Availability 


ONA is available for download at http:// 
www. freebsd.uwaterloo.ca/twiki/bin/view/Freebsd/ 
OnalInstallation [23]. Documentation is available at http:// 
www.freebsd.uwaterloo.ca/twiki/bin/view/Freebsd/ 
OpenNetworkAdministrator [24]. 
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Appendix 1: Example E-Mail Report 
Subject: ona - 1 admin changes, 2 reboots, 4 port changes, 10 config lines 


1 admin changes 

2 switches have rebooted 

4 port changes 

2 switches 10 config lines changed 


AAT TT TTT EE EEE EE ttt ttt ttt 
1 admin changes... 


#707 Jul 11 2005 09:52:21 vic UPDATE devices 
SET ‘ipname‘=aco-swhhll WHERE ‘ipname‘ LIKE aco-swhhl 


Se ttt ttt tet ttt ttt ttt ttt ttt pe 
2 switches have rebooted... 


aco-swhhll last reboot was at Mon Jul 11 6:56:37 
dccore-exsw05 last reboot was at Mon Jul 11 10:23:27 


Se tet tee ee ttt ttt 


4 port changes... 


seq date/time userid switch port setting value 
#26378 Jul 11 2005 09:58:14 bee aco-swpas6 Fe0/7 speed/duplex 100/full 
#26453 Jul 11 2005 14:13:20 peregi dccore-exsw20 5:4 description 5:4 ecenet 
#26454 Jul 11 2005 14:13:20 peregi dccore-exsw20 5:4 comment DC-3579 A34 29 
#26455 Jul 11 2005 14:13:21 peregi dccore-exsw20 5:4 untaggedvlan 90 


RE ttt tt tt tt tt ttt tt tt 
2 switches 10 config lines changed... 

## aco-swpas6 changes #3; <'T 

Last configuration change at 18:56:25 EST Sat Jun 18 2005 


NVRAM config last updated at 15:52:54 EST Thu Jul 7 2005 
NVRAM config last updated at 08:58:11 EST Mon Jul 11 2005 


version 12.0 
no service pad 
switchport access vlan 64 
! 
interface FastEthernet0/7 
+ duplex full 
+ speed 100 


switchport access vlan 64 
! 


interface FastEthernet0/8 


## dccore-exsw20 changes +3 -3 


- ## Alpine3808 Configuration generated Fri Jul 8 22:43:08 2005 
+ # Alpine3808 Configuration generated Mon Jul 11 22:46:58 2005 


- configure vlan "VLSINet" add port 5:4 untagged 
+ configure vlan "ECENet" add port 5:4 untagged 


configure port 5:4 display-string "5:4 vlsinet" 
+ configure port 5:4 display-string "5:4 ecenet" 
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NAT’d and Nested iptables Firewalls 
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ABSTRACT 


As firewalls have increased in power and flexibility, the complexity of configuring them 
correctly has grown significantly. An error in the firewall configuration can compromise the 
security of the system or interfere with normal network activity. The chance of an error increases 
when coordinating multiple firewalls, because the interaction between filters may hide errors more 
easily noticed on a single firewall. Firewalls on many networks use network address translation, 
which further increases the complexity of the firewall policy and creates additional opportunities 
for errors. Because errors in the firewall configuration are often extremely costly in time and 
security, system administrators need tools for verifying and debugging their firewall policy. ITVal is 
a tool for analyzing iptables-based firewalls that provides a plain English query language for 
simple firewall analysis. In this work, we describe extensions to |TVal that allow it to process 
network address translation rules and analyze multiple firewalls connected sequentially. 


Introduction 


Networks with a large number of hosts must 
defend against both external and internal intruders. 
While a perimeter firewall will block many external 
threats, it is useless against attacks from inside the net- 
work. With Trojan horses and viruses extremely 
prevalent, the problem of intrusions from internal 
hosts is growing rapidly [11]. To solve this problem, 
many system administrators complement the perimeter 
firewall with local firewalls on important internal 
hosts [10]. If the network is sufficiently large, the sys- 
tem administrator may also place additional firewalls 
between the perimeter firewall and groups of related 
hosts. The resulting architecture looks something like 
Figure 1, which depicts a network with a perimeter 
firewall, one unprotected host, two protected hosts, 
and a protected subnet. The protected hosts could be a 


mail server and a web server, while the protected sub- 
net might consist of clients in an accounting depart- 
ment with financial information that must be secured. 
The perimeter firewall can mitigate denial of service 
and other external threats, while the firewall on each 
workstation secures services that must be protected 
from an inside intruder. Usually, most of the worksta- 
tions have very similar filtering policies, which sim- 
plifies the distribution of changes to the policy, since 
the policy can be edited on a single administration 
host and then distributed across the network. One or 
more of the firewalls may also use network address 
translation(NAT) to further protect internal hosts or to 
work around the IPv4 address space problem. 


The Linux firewall system, iptables, which pro- 
vides NAT and stateful filtering, is well-suited for 
securing large networks of workstations and can be a 
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Figure 1: Common firewall architecture for defeating insider threats. 
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cheap solution for providing security against internal 
threats. Unfortunately, iptables rule sets are particularly 
difficult to understand and debug. Changes to the fire- 
wall may introduce subtle errors that disrupt normal 
traffic and can be difficult to diagnose. 


The multiplicity of firewalls, in networks like the 
one described, greatly increases the difficulty of 
avoiding configuration errors. Removing a rule at the 
perimeter often means exposing hosts that are not suf- 
ficiently protected by their local firewalls. Incorrectly 
adding rules to a local or intermediate firewall can 
unintentionally block important network services. 
Firewalls with NAT are even more complex because a 
set of translation rules must be considered in addition 
to the filtering rules. 


For example, consider the filtering chains given 
in Figure 2 that could be used to secure the network 
shown in Figure |. The first chain is the forwarding 
chain of a perimeter firewall that protects a network 










[Target] prot] source [destination | flags | 
PT [prop | tep [113.137.9.024 | anywhere |__| 
113.137.4103 |__| 
tp 
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113.137.10.0/24 from intrusions on an insecure net- 
work 113.137.9.0/24. Rule 1 of the chain blocks traffic 
from the insecure network. Rule 2 protects the mail 
server by blocking all traffic from the outside world. 
The remaining rules secure various services that 
should be allowed to pass through the firewall. The 
second chain is the INPUT chain of the internal mail 
server, 113.137.10.3. It permits SMTP, secure IMAP, 
and SSH traffic, but blocks anything else. 


There are several invariants that the administrator 
wishes to preserve about this network. First, web traffic 
from anywhere but the insecure network should always 
be allowed to host 113.137.10.4, the web server. Sec- 
ond, hosts from the outside world should never be able 
to SSH into the mail server. Third, no traffic should 
ever be permitted from the insecure network. 


Let’s assume that the administrator decides to 
allow SSH traffic through the perimeter firewall for 
hosts on subnet 113.137.8.0/24 by adding a rule to the 






Rule Set on the Perimeter Firewall 













ACCEPT 


[target | pro 
cr [ACCEPT | tep 
icp 






anywhere TCP dpt:25 


Rule Set on the Mail Server 


Figure 2: Rule sets for an example network. 
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Target [prot 
PT pRoP_ | tep | 113.137.9.024 | anywhere | _——_—| 
it tep_[113.137.8.024 | anywhere | —_—| 
S| DROP | tep [anywhere | 113.137.103 |__| 

ip 









Figure 3: An incorrect perimeter rule set. 
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Chain FORWARD (Default Policy DROP) 


[oo arget [prot | source [destination | flags | 
oT DROP | tep | 113.137.9.024 | anywhere |__| 
Pa [DROP | tep | anywhere | 113.137.103 |__| 
tep | 113.137.8.024 | anywhere [|__| 
ip 


TCP dpt:53 


Figure 4: A correctly modified rule set. 
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forwarding chain of the perimeter firewall as shown in 
Figure 3. The first rule and the last three rules are the 
same as those in Figure 2, but the second rule is new. 
This change preserves the first and third invariant, but 
violates the second, because SSH traffic from the out- 
side world can now reach the mail server. To correct 
the violation, she can either add restrictions to the fil- 
ter on the mail server or switch the order of rules two 
and three in the perimeter filter. 


A correct rule set for the perimeter firewall is 
shown in Figure 4. The new rule set allows HTTP and 
DNS traffic from 113.137.8.0/24, but preserves all 
three invariants. 


Existing Tools 


There is a fairly large body of tools available for 
testing firewalls. Port scanners, such as nmap [6], can 
be used to reveal open and closed ports on each host. 
Tools that perform general security audits, such as 
SATAN [5], Nessus [2], SARA [17], and ISS [9] also 
include components for testing firewalls by sending 
specially crafted packets to a host. In addition, there 
are a few tools, such as Ftester [3], designed specifi- 
cally for analyzing stateful firewalls. 


All of these tools are active tools that test the 
firewall by sending traffic through it. This has the dis- 
advantage of consuming bandwidth and interfering 
with normal traffic. Furthermore, active tools are usu- 
ally very inflexible. Rather than providing general 
functionality for investigating the firewall configura- 
tion, they are usually designed to test specific vulnera- 
bilities. Also, they usually only simulate packets origi- 
nating from a single host or small group of hosts. 
Incorrectly configured firewalls that allow packets 
from an untested host will pass the test even though an 
error exists. Some tools use address spoofing to miti- 
gate this problem, but because of bandwidth con- 
straints, no active tool can test every possible address 
that might originate a packet to the firewall. 


More importantly, active tools do not work well 
with multiple firewalls and network address transla- 
tion. Because packets dropped by one firewall are 
never seen by the second, it is often difficult for an 
active tool to generate a spoofed packet that will 
exploit configuration errors in both firewalls. Also, 
replies to packets with NAT’d source addresses may 
never been seen by the active analysis tool. 


Because active tools have these drawbacks, pas- 
sive tools, which perform an offline analysis of the 
firewall can be more practical. One such tool is the 
Lumeta firewall analyzer [18], a commercial product. 
Lumeta is based on Fang [1] and provides general 
query capability for Checkpoint and PIX firewalls. 
Because Lumeta provides an offline analysis of the 
firewall policy, it has many advantages over active 
tools. Unfortunately, Lumeta is not designed to work 
with iptables firewalls. 
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A few other groups have also done some work 
on passive analysis. A team at the University of Texas 
has developed a tool that uses SQL-like queries for 
firewall analysis [12]. They have also developed a sys- 
tem for improving the structure of the rule set using 
decision diagrams [7]. Another group has used deci- 
sion diagrams to implement basic firewall queries [8, 
16]. None of these tools are specifically targeted at ipt- 
ables and they do not support NAT. 


In previous work, we presented an open source 
tool, [TVal, for performing a passive analysis of a sin- 
gle iptables firewall. ITVal uses an efficient decision 
diagram library [13] to provide a plain-English query 
language that a system administrator can use to 
quickly test for vulnerabilities. [TVal is particularly 
useful for evaluating changes to the firewall configu- 
ration. A system administrator can perform an |TVal 
audit before and after each change to the firewall and 
examine the results to quickly determine if any of the 
important security invariants of the network have 
changed. As presented in [15], [TVal supported neither 
NAT nor analysis of multiple connected firewalls. In 
this work, we present extensions to |TVal that allow it 
to take these into account. 


ITVal 


ITVal implements a query engine for evaluating 
the configuration of a firewall. Some example queries 
are shown in Figure 5. The main components of each 
query are the keyword QUERY followed by an 
optional input chain, a subject, and a query condition. 
The input chain parameter determines whether the 
query should consider traffic inbound to the firewall, 
traffic outbound from the firewall, or traffic forwarded 
through the firewall. 


The subject tells [TVal what information to report. 
For instance, the subject SADDY instructs ITVal to list 
the source addresses of packets that match the query. 
The condition is made up of primitives that can be 
combined with the logical operators AND, OR, and 
NOT to form complex queries. The available primi- 
tives are FROM, TO, ON, FOR, WITH, and IN, which 
specify source address, destination address, source 
port, destination port, TCP flag status, and connection 
state, respectively. There is also a special primitive, 
LOGGED, which matches packets for which the fire- 
wall contains a matching LOG rule. The reader is 
referred to [15] for the details of the query language. 
One feature of the query language we will exploit in 
our examples is the ability to name groups of hosts 
and services for use in multiple queries with the 
GROUP and SERVICE keywords, respectively. 


The internal representation of both the queries 
and the rule sets is handled by the FDDL [13] decision 
diagram library. FDDL provides a data structure called 
a Multi-way Decision Diagram(MDD) for represent- 
ing large sets of vectors compactly. MDDs are particu- 
larly well suited for representing firewall rules. In fact, 
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an MDD implementation of the iptables filtering algo- 
rithm showed significant performance gains over the 
existing implementation [4]. 


e¢ QUERY SADDY TO 192.168.*; 
List all hosts with access to subnet 
192.168.0.0/16. 
e QUERY DPORT FROM 113.137.10.* AND NOT 
FOR TCP 993; 
List all destination ports, except the secure 
IMAP port(993), that can be accessed by hosts 
in the 113.137.10.0/24 subnet. 
QUERY SPORT NOT FROM 192.168.1.101 AND 
FOR 137.113.6.2; 
List all source ports open on host 137.113.6.2 
to machines other than host 192.168.1.101. 
QUERY DADDY FOR TCP 25 AND (IN NEW OR IN 
ESTABLISHED); 
List all hosts that can receive packets on port 
25 on a connection in the NEW or ESTAB- 
LISHED state. 
QUERY DADDY FROM 192.168.1.* AND (FOR TCP 
25 OR FOR TCP 80 OR FOR TCP 110); 
List all hosts that can receive SSH, SMTP, or 
HTTP traffic from hosts on the 192.168.1.0/24 
subnet. 
Figure 5: Some example |TVal queries. 


Formally, an MDD is a directed acyclic graph in 
which the nodes are organized into levels and every 
arc from a node at a level k>0 points to a node at level 
k-1. In ITVal, each level of the MDD corresponds to 
one attribute of a packet potentially seen by the fire- 
wall. Every node of the diagram represents a set of 
packets that share some common attributes. Each arc 
at level k represents a choice of value for attribute k. 
An example MDD for the last query of Figure 5 is 
shown in Figure 6. Because we do not allow duplicate 
nodes with all arcs pointing to the same descendants, 
this means that a path through the MDD represents 
exactly one packet. 


We use MDDs to represent both the rule set of 
the firewall and a set of queries. To represent a rule 
set, we add a level of terminal nodes to the bottom of 
the MDD which correspond to the ultimate fate of the 
packet (accepted or dropped) as determined by the 
rule set. To represent the queries, we instead add a 
level of terminals that indicate whether the packet 
matches the query conditions or not. 


Intuitively, a rule set MDD represents the set of 
packets accepted by the firewall, while a query MDD 
represents the set of packets that satisfy the query condi- 
tions. When depicting MDDs graphically, we will often 
show only paths to the ACCEPT node or the MATCHES 
node. To save space, we also omit the terminal level. 


To perform a query, we first represent the query 
and the rule set as MDDs. The MDD for the rule set is 
constructed from a rule set description generated using 
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the “iptables -L -n” command. The MDD for the query 
is generated from a query file provided by the user. We 
then evaluate the query on the rule set by applying an 
MDD intersection operator to the two MDDs. The inter- 
section of the two MDDs is the set of packets accepted 
by the firewall which match the conditions of the query. 


Level 20: 

Source Address, Octet 1 
Level 19: 

Source Address, Octet 2 
Level 18: 

Source Address, Octet 3 
Level 17: 

Source Address, Octet 4 
Level 16: 

Destination Address 
Octet 1 

Level 15: 

Destination Address 
Octet 2 

Level 14: 

Destination Address 
Octet 3 

Level 13: 

Destination Address 
Octet 4 

Level 12: 

Protocol 

Level 11: 

Source Port, Byte 1 


Level 10: 
Source Port , Byte 2 





Level 9: 
Destination Port, Byte 1 


Level 8: 
Destination Port, Byte 2 


Level 7: 
FIN Flag 
Level 6: 
SYN Flag 
Level 5: 
RST Flag 
Level4: 
PSH Flag 
Level 3: 
URG Flag 
Level 2: 
ACK Flag 
Level 1: 


v7 
Connection State Hatablished 


Figure 6: An example MDD. 


In the following sections, we present the MDD- 
based infrastructure of the changes made to ITVal to 
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provide support for nested firewalls and NAT. The 
system administrator should have access to such infor- 
mation in order to increase his understanding of, and 
trust in, the tool. 


Composing Nested Firewalls 


In order to extend ITVal to work with multiple 
firewalls, we introduce the concept of a meta-firewall. 
A meta-firewall is an imaginary firewall that repre- 
sents a composition of the rule sets of two or more 
serially connected firewalls. To analyze a meta-fire- 
wall, the user passes the names of the rule set descrip- 
tion files on the command line. The order of the file- 
names must reflect the topology of the firewalls, with 
the innermost filter first on the command line and the 
outermost filter last. 


The meta-firewall has three filter chains analo- 
gous to the three built-in chains of a normal firewall. 
The FORWARD chain of the meta-firewall regulates 
traffic passing through all the firewalls in either direc- 
tion. The INPUT chain of the meta-firewall regulates 
traffic inbound to the innermost firewall through all of 
the outer firewalls. The OUTPUT chain represents 
traffic generated by the innermost firewall that suc- 
cessfully passes through the outer firewalls to the out- 
side world. 


Queries are performed against the meta-firewall 
as if it were a single iptables firewall. For instance, the 
query 

QUERY FORWARD DPORT FOR 192.168.* 
AND IN NEW; 
will list the destination ports of packets bound for the 
192.168.0.0/16 subnet that pass through all the fire- 
walls in the set. 


To construct the meta-firewall, [TVal joins the 
MDD for each chain of the component filters using the 
MDD intersection operator described in [15]. The algo- 
rithm is recursive and uses caching to improve perfor- 
mance. Using the caches, we are guaranteed to consider 
each pair of nodes in the MDDs only once. Because the 
performance depends only on the number of nodes in 
each MDD and not on the number of rules, this algo- 
rithm scales well to extremely large rule sets and 
queries. 

Pseudocode for generating the meta-firewall is 
shown in Figure 7. The INPUT chain of the meta-fire- 
wall is constructed by intersecting the FORWARD 


Firewall* ConstructFirewall(int n, Firewall* 
1 newFW = NewFirewall() 
newFW.forward = fws[0].forward 
newFW.input = fws[0].input 
newFW.output = fws[0].output 


for iin / to u-l: 


newFW.input = IntersectMDD(K, 
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return newFW. 


newFW.forward = IntersectMDD(K, 
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chains of the outer n-/ firewalls and the INPUT chain 
of the innermost firewall. The OUTPUT chain is cre- 
ated by intersecting the OUTPUT chain of the inner- 
most firewall with the FORWARD chains of the outer 
n-1 firewalls. The FORWARD chain is the intersection 
of all n FORWARD chains. 


An MDD depicting the meta-firewall for the rule 
sets in Figure 2 is shown in Figure 8. In order to save 
space, only the levels for source address, destination 
address, and destination port are shown. 






Source 
Address 1 


Source 
Address 2 


Source 
Address 3 


Destination 
Address 1 


Destination 
Address 2 


Destination 
Address 3 


Destination 
Address 4 


Destination 
Port 1 


Destination 
Port 2 


Figure 8: Combining two rule sets into a meta-fire- 
wall. 


Figure 9 illustrates how ITVal might be used to 
detect the errors described in section |. We depict the 
results of three queries before and after the incorrect 


fws) 


newFW.forward. fws[i].forward). 
newFW.input, fws[i].forward). 
newFW.output = IntersectMDD(K, newFW.output, fws[i].forward). 


Figure 7: Algorithm for constructing a meta-firewall. 
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change. The original, valid, results are shown in Roman 
font, while the later query results are shown in bold. 


Each query corresponds to one invariant that the 
administrator wishes to preserve. The first query asks 
which hosts, other than those on the insecure net, can 
access the web server. In the original results, we see 
that, as expected, any other host can access the web 
server. In the modified results, we see that this impor- 
tant invariant still holds. 


The second query asks whether the SSH port on 
the mail server can be accessed from outside the fire- 
wall. In the original results, no external machine can 
reach it. After the modification, however, the results 
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show that the SSH port can be accessed from outside 
the firewall. SSH traffic from 113.137.8.0/24 can now 
reach the mail server. By comparing these results, the 
administrator will realize that she has made a mistake 
and take steps to correct it. 


The last query tests whether services on the mail 
server are available to the insecure network. In both 
cases, the answer is no. 


Network Address Translation 
In addition to filtering packets that pass through 
the firewall, iptables provides a mechanism for modi- 


fying the destination or source address of a packet 
before and after filtering, respectively. This network 


>ITVal Example.fw mail.rs mail.nat perimeter.rs perimeter.nat 


#First invariant: Web traffic not from insecure net 


#tean always reach the web server 
GROUP insecure 113.137.9.*; 


QUERY SADDY INPUT FOR 113.137.10.3 AND NOT FROM insecure AND FOR TCP 80; 


## Addresses: [0-112].*.*.* 


[114-255] <*.*. 


113. [O-1396))."5* 


## 113. (138-255].*.* 113.137.[0-8].* 113.137. [10-255].* 


## 4294967040 results. 
# Addresses: [0-112].*.*.* 


[114-255] .* 3? 


113% [O-1361 .*.* 


# 113..(138-255].*.* 123.137.[0-8].* 113.137..[10-+255] .* 


# 4294967040 results. 


##Second invariant: External hosts should never be able to SSH to the mail 


i# server. 
GROUP internal 113.137.10.*; 


QUERY DPORT INPUT NOT FROM internal AND FOR TCP 22 AND TO 113.137.10.3; 


# Ports: 
## 0 results. 
# Ports: 22 


# 1 results. 


#Third invariant: No traffic from the insecure network can reach 


##Hthe mail server. 
QUERY DPORT INPUT FROM 113.137.9.*; 


# Ports: 
# 0 results. 


# Ports: 
# 0 results. 


Figure 9: Query results before and after the change. 





| all | anywhere | 113.137.10.101 | TCP dpt:2002 to:192.168.1.2:22 

all [anywhere | 113.137.10.101_| TCP dpt:2003 to:192.168.1.3:22_| 
Tall_[ anywhere | 113.137.10.101_| TCP dpt:2004 to:192.168.1.4:22_| 
[_all_| anywhere | 113.137.10.101 | TCP dpt:3000 to 192.168.1.2:9999 | 













NAT rules for the intermediate firewall 





Chain FORWARD (policy ACCEPT) 
[[otarget_[ prot [source | destination | flags 
}1 | DROP | tcp | 113.137.10.4 


192.168.1.0/24 | | 


Filter rules for the intermediate firewall 
Figure 10: Rule set of a NAT’ ing firewall. 
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address translation (NAT) can also alter the destination 
and source ports of the packet. Properly handling NAT 
in the query engine is important, because the modified 
packet may be treated differently by the filtering rules 
than the original packet. In order for our queries to 
take NAT into account, we must modify the rule set 
MDD to reflect each of the NAT rules. 


Like filtering rules, NAT rules are specified using 
chains. Destination NAT(DNAT) rules for incoming 
packets are specified in the PREROUTING chain, 
which is processed before filtering. In addition to the 
PREROUTING chain, iptables provides a POSTROUT- 
ING chain, processed after filtering, which is appropri- 
ate for source NAT(SNAT), and an OUTPUT chain for 
performing DNAT on locally generated packets. 


An example rule set for a NAT’ing firewall, 
which might represent the intermediate firewall in 
Figure | is shown in Figure 10. This firewall protects 
an internal network 192.168.1.0/24 by hiding the 
addresses of hosts from the outside world. To access 









Chain FORWARD 


-—[oarget_[ prot [source | 
DROP 113.137.10.4 | 192.168.1.0/24 | 
DROP 113.137.10.3 | 113.137.10.101 | TCP dpt:3000 


policy ACCEPT) 
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a host, an external system must connect to the 
NAT?’ ing firewall, which will forward the connection 
on the appropriate port. Each NAT rule has a domain 
and a range. The domain of the NAT rule specifies 
the set of packets that will be modified. In Figure 10, 
the domain of the first rule is ‘‘all TCP packets from 
113.137.10.101 on port 2002’. The range of the NAT 
rule specifies a set of new destination or source 
addresses and ports. In this case, the range of the rule 
is “for 192.168.1.2 on port 22”’. Because NAT can be 
used for primitive load balancing, the range may be a 
set of destinations rather than a single value. In this 
case, packets may be sent to any of the addresses in 
the range. 

In the figure, the PREROUTING chain maps 
ports 2002-2004 to the SSH ports of internal hosts and 
also maps port 3000 to some proprietary database soft- 
ware running on port 9999 of machine 192.168.1.2. 
The FORWARD chain blocks traffic from the web 
server to those hosts. 







Figure 11: Incorrectly configured filter on the NAT’ ing firewall. 
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Figure 12: Applying DNAT to the example rule set. 
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To access host 192.168.1.2 in the example a user 
should connect to port 2002 on 113.137.10.101. The 
firewall will then replace the destination address of the 
packet with “192.168.1.2” and the destination port 
with “22” before passing it through the filter rules 
and sending it to the router. 


NAT adds another layer of complexity to the 
configuration of a firewall. One common mistake is to 
add filtering rules for the original address rather than 
the NAT’d address. For instance, if the system admin- 
istrator decides to further restrict access to the internal 
hosts, she might add rule 2 of Figure 11 to prevent the 
mail server from accessing the proprietary database. 
While this blocks connections to the firewall itself, it 
does not block forwarded connections to the internal 
hosts. Unfortunately, such a mistake is very subtle and 
difficult to catch. 


To model network address translation using 
MDDs, we create an operator which takes as inputs a 
NAT rule and the MDD representation of a set of 
packets. It produces as output an MDD representing 
the NAT-modified set of packets. Figure 12 shows the 
filtering rule set of the example before and after appli- 
cation of the DNAT algorithm. Tracing the MDD from 
the root node along the path for a packet from address 
113.137.10.3 to port 3000 of host 113.137.10.101, we 
find that the packet will be accepted by the firewall. 
Pseudocode for performing DNAT on the MDD repre- 
sentation of the rule set of a single firewall is shown in 
Figure 13. SNAT is implemented similarly. 


The algorithm takes as parameters a NAT rule 
and two MDD nodes at level k. The first node repre- 
sents a set of unmodified packets. The second node 
represents the same packets after NAT has been 


node_idx DNAT(NAT_RULE nr, level k, node p, 


1 if k<DPORTLEVEL: 
2 return q. 

3 if r = (k,p.q) is in the cache: 
4 return r. 

5 r = NewNode(k). 

6 If k>DADDYLEVEL: 

7 for each arc i <k, p>: 
8 
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applied. Initially, both parameters point to the root 
node of the rule set MDD. Our goal is to create a new 
set of rules which map each original packet to the tar- 
get of its NAT’d equivalent. 


Lines | and 2 check for the base case condition. 
If we have passed the levels which correspond to the 
destination port, we don’t need to do any more work 
as all the NAT related information is contained in the 
preceding levels. We simply return the node represent- 
ing the destination of the NAT’d packets. 


Lines 3 and 4 check to see if the result has 
already been computed. If so, it is retrieved from the 
cache and returned. 


Line 5 creates the result node. If the recursion 
has not yet reached the destination address levels, 
lines 6-11 set its arcs to the result of the recursive call. 
Otherwise, we consider each value 7 of the current 
attribute. If 7 is in the domain of the NAT rule, line 17 
creates an arc to child of the NAT’d node. Otherwise, 
we point arc / at the child of node p. 


Finally, in lines 20 and 21, we remove the node 
if it duplicates another node. If it is a duplicate, the 
index of the original node is added to the cache. Oth- 
erwise, the index of the result is added to the cache. 


Figure 14 provides a query that might be used to 
detect the error in Figure 11. The group “‘insecure”’ is 
a list of hosts that should be prevented from accessing 
the secure server. A correctly configured firewall 
should always return an empty result. After making 
the incorrect change to the filtering rules, the system 
administrator adds 113.137.10.3 to the group. Now, if 
she again runs the query, she will see the results in 
Figure 15 and detect the error. 


node q) 


<k, rc>{i] = DNAT(nr, k-1, pchild, pchild). 


9 r = RemoveDuplicates(r). 
10 Add (k, p, gq) = r to the cache. 
La return r. 
12 For each arc i € <k, p>: 
13 pehild = <k, p>[i] 
14 if i © Range(nr[k]): 
15 for each value j in nr[k] (i): 
16 qchild = DNAT(nr, k-1, pchild, <k, p>[j]). 
17 <k,r> [i] = Union(<k,r>[i], 
DNAT(nr, k-1, pchild, qchild)). 
18 otherwise: 
19 <k,r>[i] = pchild. 
20 r = RemoveDuplicates(r). 


21 Add (k,p.q) = r to the cache. 
22 return r. 


Figure 13: Algorithm for destination NAT of MDDs. 
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GROUP insecure 113.137.10.3; 
QUERY SADDY INPUT FROM insecure 
AND FOR TCP 3000; 


## Addresses: 

## O results. 

Figure 14: Query for detecting errors in the NAT’ ing 
firewall. 





GROUP insecure 113.137.10.3 
11:3). 13% sLO.As 


QUERY SADDY INPUT FROM insecure 
AND FOR TCP 3000; 


dt Addresses: 113.137.10.4 
df 1 result. 


Figure 15: Results for an incorrectly configured fire- 
wall. 


Nested Composition with Network Address Trans- 
lation 


In order to analyze a network that contains 
nested and NAT’d firewalls, the user specifies the file- 
names of filter rule sets and NAT rule sets alternately 
on the command line starting with the innermost fire- 
wall and working toward the outside. 


The pseudocode in Figure 16 combines NAT 
with analysis of multiple firewalls. The procedure 
DNAT_ALL applies the chain of DNAT rules pointed 
to by its first parameter to the MDD specified by the 
second parameter. The procedure SNAT_ALL works 
similarly for SNAT. 


In order to correctly derive the output chain of 
the meta-firewall, we work from the outermost fire- 
wall toward the innermost firewall combining pairs of 
firewalls. We DNAT the outermost firewall, then enter 
a loop in which we intersect the result with the 
unNAT‘d filter rules of the next firewall to be consid- 
ered. In each iteration, of the loop, we perform SNAT 
on the result of the intersection using the SNAT rules 
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of the first firewall. We then DNAT using the DNAT 
rules of the second firewall. This alternating behavior 
simulates the traversal of a packet first through the 
PREROUTING chain, then through the filtering rules, 
and finally through the POSTROUTING chain. 


To derive the input and forward chains, we per- 
form the same operations in reverse order, working 
from the innermost firewall to the outermost firewall. 


Conclusions and Future Work 


Although modifying rule sets can be a compli- 
cated and error-prone process, the use of passive anal- 
ysis tools can greatly simplify the task. Tools like ITVal 
that identify firewall configuration errors can prevent 
subtle mistakes from compromising the long-term 
security of a network. With our modifications, ITVal 
can be successfully used to analyze NAT’d firewalls in 
a hierarchical network topology. While this works well 
for networks in which the internal workstations share a 
common policy, developing algorithms for processing 
a more general topology is desirable. 


In addition to supporting destination and source 
NAT, iptables provides two special case NAT targets. 
The REDIRECT target rewrites the destination 
address of a packet so that it will be routed to the fire- 
wall itself. The MASQUERADE target rewrites the 
source address of a packet so that it appears to have 
been originated by the firewall. The REDIRECT and 
MASQUERADE targets are extremely useful for envi- 
ronments in which addresses are assigned dynami- 
cally, since the address of the original host need not be 
known apriori when designing the rule set. In order to 
represent REDIRECT and MASQUERADE tules, 
ITVal needs to lookup the IP address of the host and 
perform SNAT or DNAT using the host IP as the new 
IP address. Although this is not currently implemented 
in the tool, we plan to add it soon. In order to perform 
MASQUERADE and REDIRECT, ITVal needs to 
know the IP addresses of each network interface. 





Firewall* NAT(int n, Firewall** FW) 
1 newFW = NewFirewall() 


newFW.forward = DNAT_ALL(fws[0].dnat, 


fws[0].forward). 


newFW.input = DNAT_ALL(fws[0].dnat, fws[0].input). 
newFW.output = DNAT_ALL(fws[n-1].nat, fws[n-1].output). 


newFW.forward = IntersectMDD(K, 


newFW.forward, fws[i].forward). 


newFW.forward = SNAT_ALL(fws[i-1].snat, newFw.forward). 
newFW.forward = DNAT_ALL(fws[i].dnat, newFW.forward). 


2 
3 
4 
5 £OY 4: Gn. L te Aor 
6 
7 
8 


9 newFW.input = IntersectMDD(K, newFW.input, fws[i].forward). 
10 newFW.input = SNAT_ALL(fws[i-1].snat, newFw.input). 
11 newFW.input = DNAT_ALL(fws[i].dnat, newFW.input). 


12 newFW.output = IntersectMDD(K, newFW.output, fws[(n-i)-1].forward). 
13 newFW.output = SNAT_ALL(fws[(n-i)].snat, newFw.output). 
14 newFW.output = DNAT_ALL(fws[(n-i)-1].output, newFW.output). 


15 newFW.forward = SNAT_ALL(fws[n-1].snat, newFw. forward). 
16 newFW.input = SNAT_ALL(fws[n-1].snat, newFw.input). 
17 newFW.output = SNAT_ALL(fws[0].snat, newFw.output). 


18 return newFW. 


Figure 16: NAT with multiple firewalls. 
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Since this information cannot be determined from the 
rule set, we will need to create an input mechanism for 
specifying these addresses. 


Improving the output mechanism could also 
greatly improve the tool. Queries that return a large 
number of results can sometimes generate “informa- 
tion overload.”’ Investigating ways to present this 
information more concisely, perhaps through a graphi- 
cal tool, would greatly improve the utility of the tool. 
While system administrators can use |TVal to catch 
configuration errors, it is still sometimes difficult to 
identify the particular rules that cause the problem. 
Labeling arcs in the rule set and result MDDs could 
address this issue. Identifying the rules that cause a 
configuration error would also make it possible to per- 
form guided repair of the firewall. Future versions of 
ITVal may not only catch firewall errors, but give a 
system administrator suggestions about how to change 
those errors to satisfy safety and liveness invariants of 
the network. 


The latest version of ITVal is available on the web 
at http://itval.sourceforge.net/ . 
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ABSTRACT 


Network and system administrators need to analyse network traffic for maintenance, security, 
and planning purposes. The volume of data on modern networks, however, make such analysis 
extremely difficult using existing open source tools. In this paper we argue that administrators 
need tools that will allow them to be more aware of the state of their networks, and we describe 
our vision for tools that would support such “network awareness” by analysing and visualising 
packet aggregations that are defined by both packet headers and payloads. 


As a first step towards such tools, we have developed a library called qcap, a framework for 
packet and stream reconstruction that allows applications to tap packets at all layers of the network 
stack: from network, to transport, to the application layer. qcap is fast, able to process network data 
at speeds of 120 megabytes per second on commodity hardware; it is easy to use, providing a 
simple API that requires only a few lines of code to perform complex parsing tasks; and it is 
extensible, using BNF-like grammars to describe TCP protocols. We believe that qcap can provide 
the foundation for tools that will support greater network awareness for system administrators. 


Introduction 


The behaviour of computer networks is one of 
the great unknowns of computer science. Most net- 
work protocols are well known, the communicating 
hosts act (in some sense) on the behalf of their human 
masters, and network data are available at well known 
points in the network; nevertheless, we still cannot eas- 
ily answer the question ‘what is the network doing?” 
High volume network traffic conspires with our lack of 
protocol reconstruction tools to obscure network con- 
tent from us. Simple tasks require the construction of 
specialised software tools [15] to look for known or 
expected network events. Meanwhile, the flood of 
expected traffic obscures unexpected [8, 31] and possi- 
bly malicious traffic. The irony of this predicament 
should not be ignored: computers are information pro- 
cessors, and computer networks are designed to share 
information, and yet we do not have the means to 
understand the information processing that these 
human-made constructs are performing on our behalf. 


General curiosity is not the only reason why we 
need to know what happens on computer networks as 
there are more pragmatic reasons to care. Network 
administrators need to fully understand the resources 
they control. Data flowing across the network dictates 
how the network should expand and be optimised; the 
more resources administrators can draw upon to 
understand the network, the more informed their deci- 
sions will be. Administrators also need to know the 
nature of network traffic for security reasons: when 
utilisation changes, they must be able to understand 
what has changed, and why. Further, other areas of 
computer science would benefit from a better under- 
standing of network phenomena. For example, proto- 
col designers must understand the environment their 
protocols are to inhabit. Additionally, network security 


researchers and practitioners need to understand the 
network and the effects of attacks. 


The problem of analysing multiple high-band- 
width data streams in an online, complex environment 
exists in other contexts. For example, many 
researchers have recognised that complex displays and 
multiple alarm signals can reduce the “situational 
awareness” of pilots and other operators of complex 
equipment, making them more prone to errors [13]. 
Similarly, network administrators are distracted by 
voluminous logs and detailed packet dumps, all of 
which give important information, but none of which 
can be relied upon to give the salient information for a 
given situation. Members of the agent community 
have recognised that in can be important for mobile 
code to be aware of the state of available network 
resources [9]; we believe, though, that such network 
awareness is potentially even more important for 
human administrators. 


The importance of understanding network traffic 
has been recognised by others. The U. S. Department 
of Homeland Security rephrases our question into the 
term situational awareness, which it defines as “the 
ability to identify, process, and comprehend the critical 
elements of information about what is happening to 
the team with regards to the mission.” [22] The DHS 
definition can be generalised to “knowing what is 
going on around you.” [18] We use the term network 
awareness to refer to situational awareness applied to 
the area of computer networking, which we will define 
as “knowing what is happening on the network.” 


Most tools for monitoring high-bandwidth net- 
work connections analyse netflow records (such as 
those described in the proceedings of VizSec 2004 [30, 
22, 6, 24, 16]), concentrating on traffic source and des- 
tination fields. Limiting enquiry to a subset of packet 
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headers makes sense for scalability reasons: at high 
data rates, packet payloads cannot be processed in a 
timely manner with commodity hardware. However, 
discarding packet payloads limits the scope of informa- 
tion available to administrators and researchers. With- 
out the ability to analyze packet payloads, it is impossi- 
ble to ascertain precise knowledge of the data crossing 
the network. To truly understand what is happening on 
a network, however, we need more than simple pay- 
load information. We must be able to reconstruct IP 
packets and TCP streams to provide our analysis tools 
with the same information available to the hosts at the 
endpoints of communications. We must then analyze 
the reconstructed data in a manner that allows us to 
detect what is happening in high level protocols, so 
that we can assign “intent” to network events, accu- 
rately saying why an event took place. 


Thus, a tool that supports network awareness will 
allow network packet headers and payloads to be ana- 
lysed and aggregated efficiently using an easy-to-use, 
responsive interface. As explained elsewhere, no such 
tool currently exists in the open source world. As a first 
step towards developing such a tool, we have designed 
and implemented qcap, a library for efficiently recon- 
structing network packets and streams, as well as 
analysing application level protocols. In the future, we 
plan to use qcap to develop user-friendly tools for 
understanding network data. Although qcap is not fast 
enough to analyze high-bandwidth data in real time, it 
is efficient enough to enable fast interaction with multi- 
gigabyte network captures using commodity hardware. 


In our recent work on mitigating network denial 
of service [5], we have been studying strategies for 
automatically constructing packet aggregates. qcap 
was inspired by the lack of tools for analysing network 
content, and providing and explanation of network 
events. Because of the general need for better under- 
standing of network behaviour, however, qcap should 
have a much wider appeal. 


The rest of the paper proceeds as follows. In sub- 
sequent sections, we refine and explore the concept of 
network awareness; we provide an overview of exist- 
ing tools that provide some degree of network aware- 
ness; we explore the possibilities network awareness 
offers us. We then present our work on qcap, a library 
for network awareness. We conclude with a discussion 
of limitations, challenges, and plans for future work. 


Network Awareness 


Network awareness is the ability to answer ques- 
tions quickly and accurately about network behaviour. 
It is a well-developed understanding of a particular 
computer network that allows a system administrator 
(for example) to easily explain its behaviour, allowing 
the administrator to make rapid, well informed deci- 
sions. Because of the vast amounts of data involved, 
we cannot expect the administrator to be aware of 
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every passing bit, but we can expect them to under- 
stand what classes of data to expect, the usual sources 
and destinations for most traffic types, and the identi- 
ties of the users involved. Under normal network con- 
ditions, we expect that an administrator should be able 
to make reasonable accurate predictions about the net- 
work state in the near future. Under abnormal condi- 
tions, the administrator should be able to quickly 
quantify and describe the abnormality. 


To help define the scope of network awareness, 
we present a series of questions whose answers pro- 
vide some improvement in network awareness. While 
these questions are not comprehensive, they provide 
an indication of the kinds of issues that we might wish 
to understand but that are difficult to answer using 
currently available tools. 

¢ Who is using the network? Many different 
entities use a network. We want to be able to 
determine who uses the network, and how they 
are using it. We are interested in different gran- 
ularities of “user’’: applications, hosts, people, 
and other networks. 
How is a host using the network? We should 
be able to determine the services a host is pro- 
viding and utilising. Services may run on non- 
standard ports, and may be tunnelled through 
other protocols. 
How do different network events relate to 
each other? Many network events occur as 
part of a larger chain of events. We can use an 
HTTP connection to illustrate: it begins with a 
DNS request, followed by an ARP (requesting 
the MAC address returned by the name server), 
followed by a TCP/IP connection to the named 
IP address. Finally, one or more HTTP requests 
are made to the requested web server. All of 
these events are causally related, and should be 
grouped together. Conceptually, we could go so 
far as to say that they are part of a single action. 
¢ How do low level protocols behave while 
being used by high level protocols? We should 
be able to gauge how IP and TCP react to higher 
level payloads, such as SMTP. 
¢ What network traffic is encrypted? It is nor- 
mal for TLS and ssh traffic to be encrypted; 
other encrypted traffic on the network, how- 
ever, could be evidence of attackers who are 
trying to conceal their actions. 
¢ What content is the network carrying? In 
order to understand network use, some inkling 
of user-level intent should be available. The 
closest we can get to judging intent via the net- 
work is by attempting to reconstruct the 
human-level activities that the network is being 
used for. As such, we need to be able to use 
user-level concepts where necessary, such as 
emails, print jobs, and uploads. 
¢ What credentials are being used on the net- 
work? Individual users may be associated with 
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multiple credentials. Where possible, we should 
be able to associate credentials with users. In 
environments where administrators have access to 
the encryption keys of users, it should be possible 
to decrypt passing traffic for further analysis. 

© What is the TCP state of all existing TCP 
streams? Stream state can be an indicator of 
malicious activity: many new streams, unclosed 
streams, or streams that are timing out may be 
indicative of abnormal behaviour. 

e What is the meaning of some bytes in a spe- 
cific stream? Given the existence of signature- 
based intrusion detection schemes [4], it may 
be useful to put signatures into a context by 
reconstructing the streams around the signature 
match, to better understand the significance of 
the region. 

e What classes of interactions exist on the net- 
work? Protocols can carry almost any type of 
data. Interactions should not be classified solely 
by the protocol(s) used, but by the content car- 
ried. For example, when dealing with emails 
that are being sent or received, it would make 
sense for POP, IMAP, and SMTP to be grouped 
together. However, some HTTP connections 
also carry email: so it would make sense for 
HTTP connections to web-based mail providers 
(such as Hotmail, and Yahoo) to be placed in 
the same class. 


Note that these questions can only be answered 
through analysis of complete packets (headers and pay- 
loads), and that such analysis requires the aggregation 
of packets using syntactic, semantic, and temporal cri- 
teria. In particular, we will need to reconstruct packet 
streams (e.g., TCP streams, UDP-based multimedia 
traffic) in order to determine context and meaning. 


While a detailed analysis of the current state of a 
network can help answer specific questions, the com- 
plexity of even small networks make such analysis 
difficult to comprehend for even the most skilled 
administrator. To accommodate this complexity, net- 
work awareness requires one to understand how net- 
work behaviour has changed over time. Because many 
changes are benign, we wish to know what looks 
anomalous about the current state relative to the past 
state. While anomaly detection in general is a difficult 
problem, the ability to classify packets more accu- 
rately should facilitate the development of improved 
network anomaly detection methods. 


Existing Infrastructure 


Although the questions described in the previous 
section are straightforward, current tools provide only 
limited support for answering these types of network 
awareness questions. Most of the tools described 
below were not designed as network awareness tools 
but have been pressed into service because of a lack of 
alternatives. Our tour of tools will start with libraries 
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and work up to full applications. Note that this list is 
not intended to be exhaustive so much as illustrative: 
it provides examples of classes of application, not 
every application in each class. 


The root of many existing packet capture tools is 
the BSD Packet Filter [23], which defines an elegant 
approach for winnowing packets based upon a textual 
predicate. The predicate is compiled into instructions 
for a tiny virtual machine, which can run in the packet 
capture device driver within the kernel. The BPF 
architecture has been widely accepted and incorpo- 
rated into the libpcap [28] packet capture library. 


In turn, libpcap and its parent application, tcp- 
dump, have spawned a number of command-line based 
open-source progeny, including tcpstat [19] and tcp- 
trace [1], that are well suited to auditing and debug- 
ging network traffic. In the more complex niches, we 
find ChaosReader [17], a command-line based stream 
reconstruction tool that is able to rebuild application 
level streams and store them as files; Snort [4], a sig- 
nature-based intrusion detection tool; and Ethereal [2], 
a graphical packet display tool. These tools are 
designed primarily as network debugging and intru- 
sion detection systems. Other tools [11, 10, 20] are 
useful for monitoring networked devices, diagnosing 
faults, and other administrative tasks. While each of 
these applications is well suited to locating known 
events, they are not well suited to providing informa- 
tion about the general state of the network. 


The next level of abstraction to consider are net- 
work awareness tools. These are explicitly designed to 
provide network administrators with some sort of pic- 
ture of the state of the network, usually for security 
purposes. An entire crop of these tools were presented 
at VizSec 2004 [22, 30, 6, 24], although older tools 
exist as well [7]. In general, most of these tools are 
graphical and use some variant of a two or three 
dimensional display to render network activity. The 
displays usually place network endpoints on two axes 
and the volume of traffic traveling between those end- 
points on the third axis. Most of these tools present 
their output as a scatterplot [22, 6, 24], although dual 
axes graphs were also used [30]. 


For the most part, the VizSec 2004 tools provide 
statistical information on data traveling between 
source and destination endpoints. The endpoints may 
take the form of one or more networks, hosts, or ports. 
The data may be presented in terms of packets, con- 
nections, bytes, or some other volumetric measure- 
ment. Although volumetric data gives some indication 
of who is talking to whom, it does provide an indica- 
tion of what is being said. Volumetric analysis does 
not provide answers to the questions we listed. We 
assert that volumetric analysis is insufficient to pro- 
vide network awareness. 


Forensic tools [3, 11] delve into the contents of 
data streams. They provide reconstructions of data 
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stream contents, often indexing it for searching, and 
support retrieval of text deemed to be of interest to 
users. They provide some idea of the classes of infor- 
mation that can be detected with full packet analysis. 
In particular, these tools can normalise network events 
into conceptual events and display those conceptual 
events grouped by type. For example, a listing of all 
discrete “login” events for the network can be pre- 
sented, indexed by the credential used, and the 
resource acquired; as can all downloads (via HTTP, 
FTP, or BitTorrent); message sends (via SMTP, IM, 
SMS, or IRC); or file access (via SAMBA, NFS, or 
DAV). They can also provide access to the payload of 
application layer streams using an appropriate ren- 
derer (such as reconstructing and playing the audio 
portion of VoIP traffic). 


While forensic tools might appear to be ideal 
tools for network awareness, their list-oriented inter- 
faces are biased towards answering specific (not gen- 
eral) questions, provide little support for correlating 
high-level semantics with low-level packet behaviour, 
and offer few mechanisms for comparisons and anom- 
aly detection. These limitations arise because these 
tools are designed to help dissect the specifics sur- 
rounding a particular incident rather than to help 
detect patterns that are not known in advance. 


Interestingly, Q1 Labs QRadar [21] already pro- 
vides a sophisticated network awareness _ tool. 
Although it provides many features that we are inter- 
ested in and provides a mechanism to answer many of 
the questions listed above, we still feel that qcap is 
necessary. Although QRadar provides many features 
for real-time network awareness, it does not appear to 
provide sophisticated visualizations, a low-level API 
for traffic analysis, or provide mechanisms for per- 
forming automated analysis of stream content. These 
are not flaws in the product, per se, but are indications 
of QI Labs target market of security officers in large 


192.168.0.1 


Hughes and Somayaji 


corporations. In contrast, qcap is aimed at the broader 
community of systems administrators and researchers 
who need to develop automated systems and custom 
visualization tools for studying network traffic. 


The Promise of Network Awareness 


To better understand the kind of applications we 
envision for qcap, here we present several visualisa- 
tion idioms that would provide network awareness by 
quickly informing a network observer of the state of 
the network. There are clearly a huge number of other 
visualisation idioms, each well suited to some class of 
information; thus, this list is illustrative, not exhaus- 
tive. qcap provides the basic operations and abstrac- 
tions that would be required to implement these visu- 
alisations efficiently. 


Protocol State vs. Time 


The intent of protocol state versus time display is 
to show how the state of two (or more) hosts change 
over time. Figure | shows a sample protocol state rela- 
tive to time. 


Figure | is ordered chronologically from top to 
bottom. Each side of the graph represents one of the 
hosts involved in the conversation. The left side is the 
initiator of the TCP connection, with IP address 
192.168.0.1. The right side is the server 10.0.1.1; here 
the term “server” is used solely to denote that 10.0.1.1 
was not the initiator of the connection. 192.168.0.1 is 
responsible for the first two packets sent, indicated with 
a right-pointing arrow angled down. Each arrow indi- 
cates an IP packet sent from one host to another; the 
height of the arrow indicates the amount of data the 
packet is carrying. 

The meaningful data carried from one host to 
another causes the protocol to change states. Each 
state is global to the protocol and shown as a coloured 
rectangle. The rectangle encompasses all packets that 


GET /images/foo.png Request 
Keep-Alive: true 


POST /cgi-bin/form.php 
name="fred", pass="999".... 
Keep-Alive: true 





| Response: 400 OK 
image/png 
foo.png 
200px by 100px 


Request 


Figure 1: Display of protocol state relative to time from HTTP view. 
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are sent and received while the protocol is in the given 
state. Each state is coloured according to type, to 
allow the user to visually group the states. 


Packets may carry information from multiple 
states. In other words, a hypothetical packet may con- 
tain three bytes, the first pushes the protocol into 
State1, the second pushes the protocol into State2, and 
the third pushes the protocol into State3. The arrow 
representing the packet would be shown as passing 
through Statel, State2, and State3 in order. 


Some of the packets exchanged on behalf of 
lower level protocols are meaningless to higher level 
protocols, so they are displayed in a different colour. 
Such packets include the TCP ACKs in Illustration 1, 
which are lightly greyed out, indicating that that do not 
carry any useful information for this protocol layer. 


The display is annotated with blocks of text to 
the right and left of the state boxes. Each annotation 
supplies extra information about the state. Information 
supplied by a host is shown under that host. This 
means that the client sent a GET, which the server 
replied to with an image; later, the client sent a POST. 


The protocol state at the top of the graph is 
intended to function as a combo box, allowing the user 
to select which level of the protocol stack they wish to 
view. Illustration 1 would therefore offer a selection 
between Ethernet, IP, TCP, or HTTP. If the interaction 
were part of a larger SOAP conversation, the SOAP 
view would be available as well. 


This visualisation type is already provided by 
some existing commercial tools [10, 20]. 


Volume vs. Time 


Figure 2 shows a traffic frequency graph that 
maps network activity to time. The x-axis is time, with 
the right-most portion of the graph being the most 
recent, and the leftmost being the oldest. The y-axis 
represents some value that changes over time. Like the 
host/aggregate vs. time graph, the traffic frequency 
graph can display one of many different y types 
(packet volume, traffic volume, average packet size, 
percentages of some total, or some attribute of a 






Packet Volume 


Time 
Figure 2: Display of traffic volume against time. 
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conversation that changes over time). The entities 
graphed are ordered vertically from least to most and 
named. 


The graph is intended to be modal. The user 
should be able to switch the aggregates being graphed 
from protocols (shown) to hosts, groupings of hosts, 
or any grouping of conversations. In addition, the user 
should be able to drill down into a class of traffic, to 
display on that with finer granularity divisions. 


Aggregate State vs. Time 


The state of a host can be viewed by watching its 
actions change over time. Figure 3 displays permuta- 
tions of the host/aggregate against time. Although any 
variable could be displayed on the vertical axis, for 
ease of explanation, we shall assume that the variable 
being displayed is that of packet volume. 


The graph is divided into five rows. Each row is 
a different mode of display that provides a summary of 
information about the entity on the left side of the row. 
The entity is shown as a glyph (either a single host in 
rows | to 4, or a subnet in row 5). Beside the glyph is 
a listing of protocols the entity is using. To the right of 
the protocol list, the state vs. time listing is shown. 
The graph displays information about the network 
against time. Note that all time-based graphs scroll 
from right to left: the rightmost data is the most recent. 


The first row shows an aggregation of values for 
a host. The host, 192.168.1.2, is producing a volume 
of data for three listed protocols (HTTP, POP, and 
SSH). The graphs are summaries for all conversations 
that the host is participating in. In other words, it may 
be involved in many POP connections, but the value 
shown is the total for all POP connections. As the “+” 
to the left of the protocol name suggests, there is a 
hierarchical listing of information that the user may 
gain access to by “opening” that protocol. 


Since there are many ways of grouping aggre- 
gated information together, rows 2 and 3 will illustrate 
three possible forms of grouping. It is our intent that 
the “+”? beside the protocol name be a means of 
cycling through the three possible views. 


Shoutcast 


SSH 


HTTP 


IPP 
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Row 2 provides a brief textual description of the 
state of each protocol the host is using. The descrip- 
tions are protocol dependent. The third row features a 
detailed summary of the HTTP traffic of 10.0.0.1. The 
only two items available for HTTP are the total num- 
ber of connections, as well as the total number of con- 
nections to unique hosts. The values displayed depend 
on the protocol being viewed: they could easily 
include the number of web pages downloaded; the 
average number of objects per web page; the hosts 
currently connected to; or the user agent performing 
the requests. 


The fourth row shows a breakdown of conversa- 
tions that the host 10.0.1.2 is participating in. It is one 
of the modes mentioned under “Row 3.” 

¢ The top line, labelled “HTTP,” shows a graph 
of the total volume of traffic for the given pro- 
tocol type; this total is the total for all of the 
conversations listed underneath it. 

e The second line shows a graph of the volume of 
traffic for the conversation 10.0.1.2 is having 
with 20.2.2.2. 

e The third line shows the conversation that 
10.0.1.2 is having with 30.3.3.3 as a listing of 
protocol states. Each state is concisely named, 
and tagged with a protocol-specific colour. If the 
state is too thin to display a name, it will simply 





192.168.1.2 





—-HTTP 
20 Connections 


12 Unique 





30.3.3.3 


10.0.1.2 
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be coloured. Because the rightmost side is the 
“newest” side, the states will be described in an 
English-readable manner. 


The fifth row shows a grouping of hosts on the 
network 192.168.*.*. Like the per-host display, each 
protocol is displayed as a graph. The protocol can be 
opened to display either a textual summary or a listing 
of all conversations it contains. 


With the aid of a context menu, the user could 
add or remove hosts from this aggregate. 


Summary 


The three visualisations described in this section 
require features not present in known open source 
libraries and tools. They include: 

¢ Indexing provides fast access to network infor- 
mation based upon queries, or walking dis- 
plays. This allows the graphical display to be 
recomputed quickly, by processing only those 
packets that are relevant to the current visuali- 
sation. 

¢ Random access into packet traces must be 
provided to allow the visualisation tool to prop- 
erly exploit the features of indexing. Even if the 
tool is designed to run online, it will have to 
keep either a rolling buffer of recent data if any 
kind of historic display is to be provided. 


+HTTP cost ce I 
+POP Mic a 
S4SSH dd nell ny eee 


20 Conversations, 2 Authenticated, | page/Smin 2 
1 Conversation, Logged in as fred@nowhere.com 


2 Conversations, using RSA, SOKB/s 


+POP __ gy me 
+SSH sed ocala 


192.168/16 +Hosts 


Figure 3: Display of aggregate state against time. Each row indicates the state of a specific entity. 
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“Recent data” in this context can refer to pack- 
ets or aggregates of traffic information that are 
stored in memory. 

e Task-specific internal models allow large data 
sets to be aggregated, manipulated, and dis- 
played quickly and efficiently. Such models 
also provide the means to create very informa- 
tive and expressive slaved visualisations [26]. 

e Access to network, transportation, and 
application layer data is necessary to provide 
a high level view of data. 

More complete discussions of the basics of visualisa- 
tion are available in [27, 26]. 


A Library for Network Awareness: qcap 


The features listed in the Summary cover a wide 
variety of computer science disciplines, from database 
maintenance and access to visualisation techniques, 
that have been thoroughly addressed elsewhere. How- 
ever, one feature is currently missing from the pan- 
theon of tools and libraries available: we have no tools 
for the wholesale decomposition of large volumes of 
packets in speeds approaching real-time. There are no 
known APIs or libraries for reconstructing conversa- 
tions from packet traces, and subsequently decompos- 
ing those conversations into their logical parts.’ We 
wish to gain access to the payload of packets to exam- 
ine the relationships between application-layer payload, 
individual packets, transportation layer interactions, 
and network events. In order to do that, we need a 
means of reconstructing application-layer streams that 
preserves network layer information. The best means 
that we can see to do that is to provide a processing 
layer that sits on top of a BPF implementation and pro- 
vide higher network reconstruction functionality. 


We have created the qcap library to address these 
needs. It is an open source library that uses an IP and 
TCP reconstruction engine derived from libnids library, 
which in turn was derived from a version of the Linux 
2.0 networking stack [29]. It can translate individual 
packets into complete conversations. Those conversa- 
tions can be tapped at any point during reconstruction, 
to allow the tool-user to fully understand the signifi- 
cance of each packet. qcap provides: 

¢ Packet parsing which allows an application to 
query fields in a packet. For example, given a 
domain name query packet, we would be able 
to query for any of the fields in it: be they IP, 
UDP, or DNS. 

e Packet reconstruction rebuilds fragmented IP 
packets. Different network stacks can recon- 
struct corrupted or malicious fragmented packets 
in different ways [25]. qcap provides a mecha- 
nism to allow an application to control how frag- 
mented packets are rebuilt, allowing qcap to 
properly emulate different network stacks. 


llibnids [29] performs stream reconstruction, but does not 
provide packet or stream decomposition. 
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e Stream reconstruction creates — stream 
“objects” from a set of packets. The stream 
objects allow the application to read a data- 
stream that is identical to that on the receiving 
end of the connection. 
¢ Stream parsing provides a means for the appli- 
cation to easily dissect a conversation. qcap 
parses the stream text, allowing the application 
to request specific syntactically defined por- 
tions of that text. 
Design of qcap 

We use libpcap as a guide for our design. It is old, 
well established, and still actively maintained. It 
appears to be the most popular open source packet 
acquisition library, used in numerous open source 
projects [28, 19, 1, 4, 2, 29]. Its design is simple, offer- 
ing a subscription interface to listen for packet arrival. 


We provide two subscription interfaces, one for 
packets, and one for portions of TCP streams. The 
packet subscription function is gqcap_packet_han- 
dler_add(), which associates the callback with a spe- 
cific stage of packet reconstruction or stream assem- 
bly. Meanwhile, TCP streams can be subscribed to 
with qcap_tcpstr_handler_add(), which causes a callback 
to be triggered when a specified syntactic element in 
an application-level stream is found. 


To support these two functions, we have three 
types of object: qcap_packet_t, qcap_tcpstr_t, and qcap_ 
tcpstr_pos_t. 

Packets 


Network level objects are instances of 
qcap_packet_t’s. Because our network level provides 
packet reconstruction for fragmented IP packets and an 
indication of logical events each packet generates, we 
add two flags to each packet: the artificial flag and the 
discarded flag. The artificial flag is used to denote 
defragmented IP packets, while the discarded flag is used 
to denote packets that are known to have been dropped. 


Our packet abstraction provides the following 
information: 

e Text is the data sent across the network layer. It 
is used for reconstruction by higher layers. 

e Arrival time when the packet was received at 
the sampling point. 

e Discarded is a flag that indicates whether or 
not the packet has been dropped before final 
delivery, either due to network state, or the 
reconstruction policy of the recipient endpoint. 

e Artificial is a flag that indicates if the packet 
was constructed artificially from other packets. 

¢ Constituents is a list of packets that this packet 
was built out of. Only artificial packets have 
constituents. 

e Fragment is a flag indicating that this packet is 
a part of another packet. 

e Processing state is the stage of processing that 
the packet is in. There are many stages, they are 
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used to indicate how the destination network 
stack is expected to treat the packet. Possibili- 
ties include: if a packet was discarded due to 
some logic error (such as a failed IP CRC 
check), if the packet triggered the creation of a 
new TCP connection, or if it is being queued 
due to TCP ordering issues. 


Each packet passes through many states as it is 
processed. The application may subscribe to 
packets entering any state. 


TCP Streams 


A TCP stream (or qcap_tcpstr_t) is an ordered 
collection of qcap_packet_t’s, in proper TCP order. 
qcap_tcpstr_t is an opaque data type, but can be queried 
with stream positions (or qcap_tcpstr_pos_t). 

TCP Stream Positions 


Stream positions are positions at exact locations 
within a stream. They are implemented as an opaque 
data type that can be copied, walked forward in the 
stream, and have the byte at their location queried. 
Given two positions in the same stream, the applica- 
tion can request all of the bytes between the positions. 


Because the qcap_tcpstr_t type is opaque and only 
queryable through qcap_tcpstr_pos_t types, it allows 
qcap to perform reference counting and garbage col- 
lection on individual packets within a qcap_tcpstr_t. 


Analysing Fields in Packets 


In addition to the callback interfaces described 
above, qcap also provides a mechanism to query fields 
from packets. In protocols that have primarily fixed- 
length fields, such as IP, TCP, and UDP headers, query- 
ing fields is trivial: it only requires byte-order conversion 
and a cast to a native type. However, other protocols 
such as DNS have arbitrary-length fields and non-stan- 
dard data encoding, meaning that an application-writer 
must write complex code to perform a simple task. 

Field querying is done with the qcap_getter_t type. 
A call to qcap_getter_compile() creates a “getter” from a 
string specification. Packets can be queried by calling 
qcap_getter_apply(). The result is converted into a partic- 
ular form, such as a string, or a boolean, and returned. 
Implementation 

qcap is written in C and is built on top of libpcap. 
Although heavily modified and extended, parts of pcap 
are also derived from libnids. At this stage, pcap does 
not use any other utilities outside of the standard plat- 
form libraries. As the bulk of the newly written code 
deals with parsing stream-based protocols, we shall 
discuss that code here. 


One of the non-functional requirements surround- 
ing qcap was that it must be easy to extend with new 
protocols. Towards this end, qcap internally uses con- 
text-free grammars that are generated from static C code. 
As it turns out, however, many protocols cannot practi- 
cally be defined solely with a context-free grammar, as 
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the protocol carries information about its own syntax. 
Such information is carried in a field providing a 
parameter, such as the length of a subsequent field, or 
a field that provides a terminating delimiter for some 
subsequent field. 


For example, consider HTTP. Well formed HTTP 
streams consist of requests (originating with the 
client), and responses (originating with the server). 
Both requests and responses can carry an optional 
“body” that can have an arbitrary length. The length 
of the body is usually defined with a “Content- 
Length” field that contains an integer indicating the 
number of bytes in the body. We could express the 
“Content-Length” statically in the HTTP context-free 
grammar, by enumerating every possible value of 
“Content-Length”, and defining the length of the sub- 
sequent body in the context-free grammar. However, 
that would result in an extremely verbose grammar. 


To avoid such large grammars, qcap implements a 
series of protocol-specific registers for each parser. As 
the parser is walking a stream, it places the value of 
specific stream elements into the associated register, 
which can be used later during the parse. Continuing 
with our example from above, gcap would store 
HTTP’s Content-Length field with a content_length reg- 
ister in the stream parser. Upon encountering the Con- 
tent-Length field, the parser would decode the associ- 
ated value as an integer and store it in the content_length 
register. Later, when the parser encounters the subse- 
quent body, it will read exactly content_length characters. 


While current grammars are built by hand, we are 
researching mechanisms for automatically generating 
the necessary static C code from protocol specifica- 
tions in Augmented Backus-Naur Form (ABNF) [12]. 


Evaluation 


In our opinion, the most interesting feature of 
qcap is the analysis of application-layer protocols. We 
believe that qcap will be most useful if it is able to per- 
form application-layer analysis at speeds approaching 
real-time. 


We have developed two test applications to test 
qcap’s speed: 

e reader opens a trace and parses it, performing IP 
defragmentation and TCP stream reconstruc- 
tion. It provides no useful output and is used for 
timing purposes. 

© ip_identity opens a trace, and gathers credentials 
sent from each IP address in the trace. Cur- 
rently, ip_identity only parses FTP usernames 
and passwords, SMTP senders, and HTTP 
authorisation requests [14]. 


Both ip_identity and valid reconstruct all IP frag- 
ments and TCP streams. qcap could be set to ignore 
TCP/IP traffic going to ports that we aren’t interested 
in, but these tests provide us with a stronger worst- 
case idea of processing time. 
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To test timing, we ran each program against the 
Lincoln Laboratory DARPA Intrusion Detection Eval- 
uation datasets [32]. The first five datasets are DIDE-1 
to DIDE-5, which correspond to the Monday-Friday 
traffic of the LBL week | traffic set, DIDE-6 to 
DIDE-10 correspond to the data gathered during week 
3. Each represents one day’s worth of traffic. 


The tests were run on an unloaded 2.8 Ghz Pen- 
tium 4 system with 1 GB of RAM, 512K of processor 
cache, and two 36.4 Gb, 10,000 RPM SCSI drives. Each 
test was run 1000 times, and the values were averaged. 


In order to minimise the amount of time spent on 
output, we ran tcpstat with an invalid BPF filter, ensur- 
ing that it would not waste too much time writing data. 
Since reader does not produce output, and ip_iden- 
tity only produces brief output at the end of the trace, 
we did not modify either application to reduce their 
volume of output. 


During our experiments, we discovered that qcap 
analyses data at a rate of roughly 2.11 microseconds 
per packet, as compared to tcpstat which analysed traf- 
fic data at a rate of 0.694 microseconds per packet. 
The difference in processing speed amounts to an 
order of magnitude: however, as shown in Table 4, 
processing times are still low: to process a 468 MB 
trace file (DIDE-6) containing 2.1 million packets 
with qcap only takes about 5.7 seconds. 


Since etherealseems to be the flagship open 
source packet processing tool, we also timed how long 
it took ethereal to open each packet trace. In our expe- 
rience, this provides a ballpark figure on how long it 
takes ethereal to perform a search. These tests were 
performed a handful of times on the same machine 
mentioned above, with the lowest timing result pro- 
vided. As shown in Table 4, ethereal is between 3 and 
20 times slower than qcap. 


In our judgment, the rates achieved by qcap are 
acceptable: we can analyze a | GB trace in roughly 10 
seconds. As we increase the number of protocols that 
we are parsing and the complexity of the protocol 
parsers, this execution time will increase; however, we 
expect the parse time to stay within the same order of 
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magnitude. In addition, the current implementation of 
qcap has not undergone any optimisation, suggesting 
that we may be able to achieve speed improvements 
with minimal effort. 


Discussion 


At this point, skeptical readers may be asking 
themselves what is new about qcap: tools already exist 
for analysing packet traces. Indeed, any of the infor- 
mation we acquire with qcap can also be acquired by 
using existing tools. For example, if a user wants to 
parse all of the cookie headers out of an HTTP conver- 
sation, they could use ngrep with a specially con- 
structed regular expression. Or, if a user wanted to 
find the contents of a TCP session, they could use 
tcpflow to pull all of the TCP sessions out of a trace, 
and then analyze them by hand or open them with 
ethereal. And general connection statistics can be gath- 
ered with much simpler tools, like tcpdump or tcptrace. 
Alternatively, snort could be used for any of the afore- 
mentioned tasks. 


Those skeptical readers should realize that qcap 
has been designed specifically for network awareness. 
It is not designed to find individual packets in a net- 
work trace, nor is it designed to find statistics on a 
specific class of event. Instead it is designed to: 

© Provide a standard interface to analyze traffic, 
regardless of protocol. 
Perform full stream reconstruction, allowing 
the application to parse strings that are split 
across multiple TCP packets, without having to 
be aware of the packet divisions. Applications 
can, however, request the information to be 
made available to them. 
Perform the drudge work of protocol syntax 
analysis, allowing the application to concen- 
trate on the meaning of the traffic. 

e Provide a simple mechanism for collecting a 

wide variety of data and statistics. 

e Handle large volumes of data quickly. 
To our knowledge, no other open-source library pro- 
vides this functionality. We have not seen an open- 
source tool that correlates large volumes of applica- 
tion-level network data. 


Processing Time (seconds) Time Per Packet (microseconds) 


Packets 


1,362,869 
1,157,328 
1,616,713 
1,807,060 


1,349,635 
2,106,744 
1,831,648 
1,849,753 
1,559,156 
1.635.425 


DIDE-6 
DIDE-7 
DIDE-8 
DIDE-9 
DIDE-10 


tcpstat | ip_identity ethereal | tcpstat | ip_identity 


69 
60 





Figure 4: Observed processing speeds of qcap compared to those of tcpstat. 
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Limitations 

While we believe qcap is quite promising, it is 
also a new library with many limitations. For example, 
because qcap builds upon libpcap, we can, for the most 
part, say that qcap shares a subset of libpcap’s limita- 
tions. There are a number of other limitations, how- 
ever, that are specific to pcap. 


First, qcap is not suited to searching for known 
strings in input text, either as a literal string, or a regu- 
lar expression. Specialised tools, such as ngrep per- 
form those tasks well. However, since qcap is a thin 
wrapper around libpcapthere is no reason why such 
tools could not be ported to use qcap. 


Next, even though qcap’s protocol parsing capa- 
bilities are well-developed, it does not deal with 
lookahead. Lookahead means scanning further in the 
stream of text being parsed to discover if any text in 
the near future would prevent the current text from 
being deemed valid. 


A trivial example involves HTTP requests. A 
valid HTTP request consists of the request-line, such 
as a “GET <url> <protocol>”’. When qcap is parsing 
the request-line, and it reaches the end of the wr/ field, 
it will inform the application that a ur/ field has been 
encountered and then continue parsing with the proto- 
col field. If the protocol field turns out to be invalid 
(because it contains the text “foo” instead of 
“HTTP/1.1,” for example), then the entire request- 
line should be deemed invalid, meaning that qcap 
should not of informed the application that an url was 
discovered earlier. 


The difficulty with lookahead is that it involves 
reading data into memory before deciding if a chunk of 
text is a part of a valid stream. If we need to perform 
lookahead to the end of a large piece of data, we could 
flood memory with data: consider an HTTP response 
that contains a 50 MB file — the response headers can- 
not be considered valid until the entire response is 
received, but that means that qcap would have to read 
the full 50 MB file into memory before deciding if it 
should accept or reject the response headers. Instead, 
we force the application developer to be aware of the 
protocol structure, and watch for failed requests. 


qcap, because of limitations in libnids, does not 
currently support any kind of parameterisation to state 
how streams should be rebuilt in circumstance not 
defined by the IP and TCP RFCs. Such situations 
include packets with invalid CRCs, overlapping pack- 
ets, et cetera. 


One potential issue for some applications is that 
qcap currently has no mechanism for synchronizing 
the two sides of a stream during parsing. Strictly 
speaking, such synchronisation is necessary to ensure 
that proper protocol state is maintained at all times. 
However, in practice we have not yet found the lack of 
synchronisation tracking to be a problem. 
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However, one of the features we feel will be nec- 
essary for any offline analysis GUI tool built on top of 
qcap is analysis of network traces larger than one giga- 
byte. In order for such an analysis to be relatively fast, 
and put a low load on the workstation, a minimum of 
data should be kept in memory; suggesting that, where 
possible, information about specific packets should be 
kept on disk, and read as necessary. To speed up over- 
all performance and searching, we assume that indices 
would be built during the initial load. 


In order for packet data to be left out of memory, 
we need a means to provide random access to a static 
trace file on disk — meaning that we should be able to 
read packets from the file in an arbitrary order. How- 
ever, libpcap does not support random access into trace 
files, meaning that solely reading specific packets 
from disk is not possible. 


There are two possible approaches to this prob- 
lem: either petitioning the libpcap maintainers to 
include random-access to libpcap files; or building our 
own file reading routines into qcap. Clearly, the first 
option is preferable. At the time of writing, we are 
engaged in petitioning the libpcap maintainers to 
include this functionality. 

Future Work 


While internal parts of the library are still evolv- 
ing, at the time of writing, the qcap API is almost com- 
plete. The existing functions are unlikely to change for 
the foreseeable future, even though new calls may be 
added. Having said this, there are are some issues that 
we hope to address in the near future. 


First, there are currently no mechanisms for 
decoding stream content. qcap should be able to 
decode either an entire stream (such as ssh or SSL), or 
portions of a stream (such as gzip-encoded HTML 
responses), in a manner that is transparent to the appli- 
cation. It should be possible for encoded regions to 
contain semantic elements that are to be recognised by 
stream parsers. Such additions would significantly 
improve the utility of qcap. 


We also plan to develop bindings to allow higher 
level languages such as Python, Perl, and Java to 
access qcap functionality. Such changes and additions 
should help facilitate the development of novel net- 
work awareness applications. The qcap distribution 
contains sample programs that provide interesting 
functionality not seen in other open source tools: 
ip_identity trawls network traces for credentials; and 
valid tests streams to see if they follow the protocol 
semantics for the ports they are using. While such 
tools can be useful, much larger scale applications are 
also possible: 

° a fast protocol debugger, along the lines of ethe- 
real, but supported by a database back-end that 
would provide fast searching and display. 

¢ anomaly-detection tools that consider the val- 
ues of individual fields in a protocol stream. 
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e “leak” detection tools that sniff passing traffic 
for sensitive content that should never leave 
hosts. 

e a fast classification tool that classifies streams 
by their purpose, either in gross terms; such as 
“exchanging email” for SMTP, POP, IMAP, 
Gmail, and Hotmail connections; or specific 
terms, such as “instant messenger conversation 
between Alice and Bob.” 


In the long term, optimisation and improvement 
of qcap will allow it to process extremely high vol- 
umes of data. Ideally, qcap will eventually be able to 
handle data at rates approaching those seen by 
medium-to-large ISPs and enterprises. When it does, 
our definition of network awareness can grow from 
today’s analysis of traffic volumes to and from hosts 
to include content-specific and aggregate analysis that 
will finally help us figure out what our networks are 
actually doing. 
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ABSTRACT 


The core of system administration is to utilize a set of “best practices” that minimize cost 
and result in maximum value, but very little is known about the true cost of system administration. 
In this paper, we define the problem of determining the cost of system administration. For support 
organizations with fixed budgets, the dominant variant cost is the work and value lost due to time 
spent waiting for services. We study how to measure and analyze this cost through a variety of 
methods, including white-box and black-box analysis and discrete event simulation. Simple 
models of cost provide insight into why some practices cost more than expected, and why 
transitioning from one kind of practice to another is costly. 


Introduction 


What is a set of “best practices’’? Ideally, it is a 
set of practices that cost the least while having the 
most value, i.e., a model of practice for which the ratio 
value/cost is maximal over the lifecycle of the equip- 
ment being managed. We have not succeeded in evalu- 
ating practices according to this yardstick, however, 
because there is no usable model of cost for any par- 
ticular set of practices. We would like a model that 
predicts, based upon particular management decisions, 
the total cost of operations that results from those 
decisions over the lifecycle of the network being man- 
aged. This is one goal of ‘analytical or theoretical sys- 
tem administration” [5, 6]. 


Many system administrators and managers con- 
sider a complete cost model to be an impossible goal 
for several reasons. First, the actual cost of system 
administration is a relatively constant and monolithic 
line item in many IT budgets; it is commonly techni- 
cally infeasible to break the lump sum cost into com- 
ponent costs for the purpose of evaluating strategy. 
Mechanisms for recording day-to-day costs (e.g., 
detailed time sheets) are often more expensive to man- 
age than the money they might potentially save. And 
for the organizations whose audit requirements force 
them to maintain detailed costing data, these records 
are usually confidential and unavailable to researchers 
outside the organization. Thus any really usable cost 
model has to be practical in not consuming resources, 
tunable for specific situations by the end-user, and 
must allow that user to incorporate confidential data 
into the model without divulging it to outsiders. 


Currently, instead of considering costs, we jus- 
tify best practices by what might best be called a 
“micro-economic” model. We say that certain prac- 
tices “*make the job easier’’, or other weak justifica- 
tions. Is “simpler” really “cheaper”? We have yet to 
prove this assertion and — in many cases — the micro- 
cosmic reasoning we use today seems to be incorrect 


at a macrocosmic (lifecycle) scale. A case in point is 
the use of automation, which is “better than manual 
changes” except that — at a macrocosmic scale — scal- 
ing increases costs in ways that are inconceivable 
when working on a small number of machines. The 
reasons for this apparent contradiction are deep and 
will be discussed later in the paper. 


Current Ideas About Cost 


The first step toward a cost model was made by 
Patterson [18], who points out that while ‘‘administra- 
tive costs”’ may be fixed and non-varying, the cost of 
downtime varies with scale of outage and disrupts 
more than computer use. Patterson’s formula for the 
cost of downtime is based upon calculation of two fac- 
tors we previously ignored as system administrators: 
revenue lost and work lost. Even if our system admin- 
istration group has a fixed size and operating budget, 
the cost of downtime varies with the severity and 
scope of outage, and lifecycle cost of operations thus 
varies with risk of outage. Patterson also points out 
that there are more subtle costs to downtime, including 
morale and staff attrition. But how do we quantify 
these components in a cost model? 


Cost modeling also depends upon risk and cost 
of potential catastrophes. Apthorpe [1] describes the 
mathematics of risk modeling for system administra- 
tors. By understanding risk, we can better make cost 
decisions; the lifecycle cost of an administrative strat- 
egy is the expected value of cost based upon a risk 
model, i.e., the sum of “cost of outcome” times 
“probability of outcome” over all possible outcomes. 


Cost modeling of risks is not simple; Cowan, et 
al. [8] point out that simply and blindly mitigating 
risks does not lead to a lowest-cost model. It is often 
better to wait to apply a security patch rather than 
applying it when it is posted, because of the likelihood 
of downtime resulting from the patch itself. Thus the 
global minimum of lifecycle cost is not achieved by 
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simply minimizing perceived risks; other risks enter 
the system as a result of mitigating the perceived ones. 


Further, Alva Couch made the bold claim at the 
last LISA [7] that the essential barrier to deployment 
of better configuration management is “cost of adop- 
tion”. The reason that configuration management 
strategies such as automation are not applied more 
widely is that it costs too much to change from unau- 
tomated to automated management. But he stopped 
short of truly quantifying cost of adoption in that pre- 
sentation, due to inadequate models. Meanwhile, many 
people pressured him in one way or another to formal- 
ize the model and demonstrate his claims rigorously. 
This paper is the first small result of that pressure. 


In this paper, we make the first step toward a cost 
model for system administration, based upon related 
work in other disciplines. We begin by defining the 
components of an overall lifecycle cost model. We look 
at real data from a trouble-ticketing system to under- 
stand the qualities of load upon a support organization, 
and discuss the problems inherent in collecting data 
from real systems. We explore the relationship between 
system administration and capacity planning, and show 
that we must determine specific rates in order to deter- 
mine costs. We borrow mechanisms for determining 
those rates from white-box and black-box cost models 
in software engineering. Finally, we turn to discrete 
event simulation in order to understand the relation- 
ships between rates and cost. As a result, we can begin 
to quantify the cost of some decisions about practice, 
including deployment schedules for new software. 
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A Simple Model Of System Administration 


First, system administration can be modeled as a 
queueing system (Figure 1) in which incoming 
requests arrive, are queued for later processing, and 
eventually dequeued and acted upon, and completed. 
Each kind of request arrives at the queue with an 
“arrival rate” and is completed in a length of time 
whose reciprocal represents a “‘service rate.” We 
embody all changes made to the network as requests: a 
request may indicate a problem or ask for a change in 
the nature of services offered. Requests arise from 
many sources, including users, management, and even 
the system administrator herself may make a note to 
change something. Likewise, requests are granted via 
many mechanisms, including work by system admin- 
istrators and work by others. 


Note that this is a more complex model than rep- 
resented by the typical helpdesk. In a typical ticket 
system, tickets represent external requests, while 
internal requests (e.g., actions generated by a security 
incident report) are not given ticket numbers. In our 
request queue, all change actions are entered into the 
queue, serviced, and closed when done. 


System administration has complex goals, so the 
request queue has a complex structure; it is (in the lan- 
guage of capacity planning [17]) a mu/ti-class queue- 
ing system consisting of a mixed set of several 
“classes” of requests (Figure 2). Many kinds of 
requests, with different average arrival rates, are com- 
bined into one request stream. Each kind of request K 
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Figure 1: System administration as a queueing system. 
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Figure 2: Multiple classes of requests and administrators. 
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has a distinct average service rate Ux (and perhaps, a 
distinct statistical distribution of service times). As 
well, a realistic system administration organization is a 
non-product system: system administrators do not 
function independently like a set of cloned web- 
servers; they communicate and interact with one 
another, affecting throughput. A product system (as in 
Cartesian product) consists of a number of compo- 
nents that function independently (so that the state- 
space of the whole system is a Cartesian product of 
the state-spaces of its parts). 


Request Arrivals 


While the overall structure of the request queue 
is complex, we observe that the structure of some 
classes of requests is easy to understand. Many classes 
of requests arrive with a ‘Poisson distribution” of 
inter-arrival times. In a Poisson distribution with 
arrival rate of A requests per unit time, 

1. The mean inter-arrival time is 1/A. 

2. The standard deviation of the inter-arrival time 
is 1/A. 

3. The arrival process is memoryless; the probabil- 
ity that a request will arrive in the next ¢ sec- 
onds is independent of whether one arrived 
recently. 


Many kinds of requests naturally obey this distri- 
bution. For example, any kind of request in which a 
large population operates independently of one another 
has a Poisson distribution, e.g., forgotten passwords. 


As well, many non-Poisson classes of requests 
(e.g., virus attacks) arrive with a Poisson distribution 
if viewed at the proper scale. While requests for virus 
cleaning of individual hosts arrive in bursts and are 
not memoryless, the arrival of the virus at one’s site is 
an independent, memoryless event. If we treat the 
virus arrival at one’s site as one event, rather than the 
thousands of requests it may generate, then new 
viruses arrive with a roughly Poisson distribution 
(because they originate from independent sources at 
relatively constant rates). Likewise, while an outage of 
a particularly busy server may generate thousands of 
tickets, the outage itself obeys a Poisson distribution 
even though the tickets resulting from the outage do 
not. Many other kinds of requests have this character; 
although actual tickets arrive in bursts, the real prob- 
lem occurs in a memoryless way that is independent of 
all other problem occurrences. Examples include hard- 
ware failures, power outages, denial-of-service attacks, 
spam, etc. 

Request Processing 


The second part of our model is how requests are 
processed. Like request arrivals, request processing 1s 
complex but there are parts of it that are understand- 
able. For example, many kinds of requests are com- 
pleted via an “exponential distribution” of service 
time. The properties of an exponential service time are 
similar to those for a Poisson arrival; if a class of 
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requests is serviced with an exponential rate of uw 
requests per unit time, then: 

1. The mean time for servicing a request is 1/1. 

2. The standard deviation of service time is I/u. 

3. The service process is memoryless; the proba- 
bility that a request will be finished in the next ¢ 
seconds is independent of whether we know it 
has been in progress for s seconds already. 


The last assumption might be paraphrased “A 
watched pot never boils.” 


Examples of requests that exhibit an exponential 
service time include password resets, routine account 
problems, server crashes, etc. For each of these, there 
is a rate of response that is independent of the nature 
of the specific request (i.e., which user) and seldom 
varies from a given average rate LL. Requests that can- 
not be serviced via an exponential distribution include 
complex troubleshooting tasks, and any request where 
the exact method of solution is unknown at the time of 
request. In general, a request for which the answer is 
well documented and scripted exhibits an exponential 
distribution of service times; requests with no docu- 
mented response do not. 


Lessons From Capacity Planning 


Real data discussed below shows that inter- 
arrival times may not exhibit a Poisson distribution, 
and that service times may not be exponentially dis- 
tributed. However, much is known about the perfor- 
mance of idealized queues governed by Poisson and 
exponential distributions, and there are many system 
administration tasks for which these performance esti- 
mates are reasonable. 


A queue that exhibits Poisson arrivals with rate 1 
and has c independent system administrators working 
with service rates Lt is called an ““M/M/c”’ queue. The 
first M stands for ‘memoryless’ (Poisson) arrivals, the 
second M stands for ‘memoryless’ (exponential) ser- 
vice times, and c is a count of servers (administrators) 
all of whom complete requests with rate LU. The behav- 
ior of an M/M/c queue is well understood and is com- 
monly used for capacity planning of server farms and 
IT infrastructure. 


For an M/M/c queue, whenever A/cu <1, the 
probability that the queue is cuapty is 
So (uy A 1 (aye (1) 
oar c! 1-A(cu) 
and the “mean time in system” (average wait) for a 
request [15] is 





_ So(A/wy l 
= ——____ + -. 
clew(l — (cu)? 

The mean time spent waiting for 7 requests to be 
serviced is n times the mean wait for one. More 
important, this equation allows us to predict whether 
adding more system administrators will not solve a 
response-time problem. As c grows, the first term of 


y 


(2) 
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the above equation goes to 0 and the response time 
converges toward the theoretical minimum 1/LL. 


Many other equations and relationships exist for 
more general queues. In this paper, we will consider 
only M/M/c models; for an excellent guide to other 
models and how to predict performance from them 
(including excel spreadsheets for decision support), 
see [17]. 

Learning From Real Data 


From above, it is easy to analyze systems that 
behave according to Poisson arrivals and exponential 
service. How realistic are these assumptions about 
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system administration? To explore this, we examined a 
ticket queue from a live site (Tufts ECE/CS). Note that 
no one knew, until very recently, that anyone was 
going to analyze this ticket data. It is thus free of 
many sampling biases. It is, however, difficult to 
determine exactly when many tickets were closed. 
This is because there is no site requirement to close 
tickets promptly, and many tickets are closed by stu- 
dent staff who monitor the ticket queue, sometimes 
long after the problem has been addressed. 


Plotting ticket numbers (an increasing sequence) 


against time (Figure 3) shows little or no evocative 
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Figure 3: Ticket durations in ECE/CS from 7/2004 to 7/2005. 
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Figure 4: Ticket durations less than 30 days. 
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patterns. Each ticket is plotted as a horizontal line, 
with beginning and end representing creation and 
completion time. The Y axis represents ticket number; 
tickets due to spam have been deleted and the result- 
ing queue renumbered as increasing integers with no 
gaps. Note particularly that several tickets are still 
open after several months. 


We discovered very quickly that there were two 
classes of service: one for short-duration requests and 
another for long-duration requests. Viewed alone, the 
requests that took less than a month exhibit relatively 
consistent response times (Figure 4). 


Request arrivals are not Poisson. For arrivals to 
exhibit a Poisson distribution, the mean of inter-arrival 
times must be equal to their standard deviation. In this 
case, the overall standard deviation of inter-arrival 
times (9580 seconds or = 2.65 hours) is about 1.37 
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times the mean (6971 seconds or = 1.94 hours), indi- 
cating that there are periods of inactivity. Looking 
deeper, Figure 5 shows one problem: arrival rates are 
not constant, but instead sinusoidal over a 24-hour 
period. In this graph, ticket arrivals are shown by hour, 
summed over the lifetime of the Request Tracker (RT) 
database. The times are corrected for daylight savings 
time, and show more intense traffic 9 am to 5 pm with 
a small dip in traffic at lunch. Ticket closures show a 
different pattern (Figure 6) with a hotspot at 3 pm that 
defies explanation, until one realizes that a student 
administrator charged with monitoring and closing 
tickets starts work at that time! 


Measured “time in system” does not seem to be 
exponential, either. If, however, one omits requests 
with time in system greater than one month, the 
remaining requests exhibit a distribution that looks 
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Figure 5: Ticket arrivals exhibit sinusoidal rate variation over 24 hours. 
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similar to exponential (Figure 7). The figure contains a 
histogram of the number of requests processed in each 
number of days. Note, however, that this figure repre- 
sents service time p/us time spent waiting in queue, so 
it cannot be used to compute an accurate service rate. 


From the data, we see that requests are multiclass 
with at least two distinct classes of requests: 

1. A vast majority of requests are resolved quickly 
(in less than one month, with a mean time in 
system of about 3.6 days). Arrival times for 
these requests seem to be governed by a sinu- 
soidal non-stationary Poisson process, 1.€., 
arrival rates seem to vary between a daily high 
and low on a sine-wave pattern. 

2. A small number of requests have an indetermi- 
nate and long time in system. Arrival times for 
these requests show no discernible structure 
(perhaps due to lack of enough examples). 

3. The average rate of ticket arrival is gradually 
increasing over time. In our case, this seems to 
be partly due to new faculty hires. 


This data also exhibits, however, the main diffi- 
culties of extracting performance statistics from ticket 
queues: 

1. Service times are recorded inaccurately because 
there is no particular advantage to recording 
them accurately. Most tickets are closed late, 
because it is not the job of the administrator 
answering the ticket to close it, but just to solve 
the problem. In our case, many tickets are 
closed by student staff some time after the 
problem has been solved. 

2. The class of a particular request is not always 
easily discernible. It is easier to group requests 
by time of service rather than class of problem. 
In our case, there is a clear distinction between 
requests for which an appropriate response is 
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documented, and those for which an appropri- 
ate response is unknown. The former take on 
average the same time to resolve, while the lat- 
ter vary widely. 

3. Emergent patterns in the data are only obvious 
if one is very careful about correcting for envi- 
ronmental issues. For example, data did not 
exhibit a sinusoidal arrival rate until it was cor- 
rected for daylight savings time (DST)! 

4. Ticket data does not indicate the severity of a 
problem. There are no discernible “flurries” or 
“bursts” of data for severe problems; often 
only one or two people bother to report a major 
outage. 


Other practitioners have mentioned that there are 
several ways that request queue data can be biased by 
operating policy. 

1. If people are rewarded for closing tickets 
quickly, they tend to close them early, before an 
actual resolution. 

2. If people are rewarded for only the tickets they 
choose to resolve, obviously difficult tickets 
will be avoided. 


The final issue plaguing the use of real request 
queue data is privacy. Real request data contains flaws 
in practice. For example, some requests for which 
there should be documented scripts remain undocu- 
mented, some requests are forgotten, and some 
requests can take an embarrassing amount of time to 
resolve. For this reason, it is difficult for researchers to 
get real data on the nature of requests and their service 
times, for sites other than their own. 


One lesson learned from our data is the power of 
good documentation. If an appropriate response to a 
problem is documented or otherwise well known, 
there seems to be no significant difference in response 
time invariant of the nature of the problem. It is 
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Figure 7: A histogram of the frequency of tickets resolved in each number of days has a near-exponential shape. 
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surprising that to a first approximation, differences in 
service times for subclasses of requests do not emerge 
from the data. One possible reason for this effect is 
that in these cases, communication time with clients 
may dominate the time needed to solve the problem 
once it is clearly defined. 


Conversely, problems with no documented 
response wait longer and may never be solved. At our 
helpdesk, student staff solve routine problems and 
defer only problems with no documented solution to 
second-level triage staff. Since the second-level staff 
are often planning or deploying new architectures, 
requests without documented solutions await their 
attention and compete with deployment tasks. Of 
course, once solved and documented, such a request 
becomes quickly solvable. 


In our data, a simple pattern emerges. System 
administration is composed of a combination of rou- 
tine tasks and complex troubleshooting and discovery 
that borders upon science. Our site’s practice is per- 
haps best described as a two-class queueing system, 
with a large number of routine requests with docu- 
mented and/or known solutions, and a smaller number 
of requests requiring real research and perhaps devel- 
opment. For the most part, the routine requests are 
accomplished by system administrators acting inde- 
pendently, while more complex requests may require 
collaboration between system administrators and take 
a longer, relatively unpredictable time to complete. 


A Simple Model Of Cost 


Given the above model of system administration 
as a queueing system, we construct a coarse overall 
model of cost, based upon the work of Patterson [18] 
with some clarifications. 


First, cost can be expressed as a sum of two com- 
ponents: the “cost of operations” and the “cost of 
waiting for changes.” The “cost of operations” con- 
tains all of the typical components of what we nor- 
mally consider to be cost: salaries, benefits, contracts, 
and capital expenditures such as equipment acquisi- 
tion. For most sites, this is a relatively predictable cost 
that remains constant over relatively long time peri- 
ods, e.g., a quarter or a year. The “cost of waiting” is 
a generalization of Patterson’s “cost of downtime”, 
that includes the cost of waiting for changes as well as 
the cost of waiting for outages to be corrected. 


While the cost of downtime can be directly cal- 
culated in terms of work lost and revenue lost, the cost 
of waiting for a change cannot be quantified so easily. 
First we assume that R represents the set of requests to 
be satisfied. Each request r € R has a cost C, and the 
total cost of waiting is 

Cy = XC... (3) 

reR 
We assume that for a request 1 (corresponding to either 
an outage or a desired change in operations), there is a 
cost function c,(f) that determines the instantaneous 
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cost of not accomplishing the change, and times 4,; and 
t., at which the change was requested and accom- 
plished. Then the tangible cost of waiting is the integral 
(running sum) of c,(t) over the waiting period: 
fa 
C.= J en(oat. (4) 
ty 
If as well c,(t) is a constant 
C, ms (t, sai t Jc. = 1,C, (5) 
as in Patterson’s paper. In general, this may not be 
true, e.g., if the change reflects a competitive advan- 
tage and the effects of competition become more 
severe over time. For example, in the case of security 
vulnerabilities, vulnerability is known to increase over 
time as hackers gain access to exploits. 


System administrators control very little of the 
process that leads to lifecycle cost, but the part they 
control — how they work and accomplish tasks — can 
partly determine the cost of waiting. In this paper, we 
consider the effects of practice upon the cost of wait- 
ing in a situation in which the budget of operations is 
held constant over some period, e.g., a quarter or a 
year. Situations in which cost of operations can vary 
(e.g., by hiring, layoffs, or outsourcing) are left for 
later work. 


The cost function c,(f) must model both tangible 
(work lost) and intangible (contingency) factors. For 
requests concerning downtime, the cost of waiting 
may be directly proportional to work and revenue lost, 
while for requests involving enhancements rather than 
downtime, work lost and revenue lost can be more dif- 
ficult to quantify. Also, the costs of waiting for 
enhancements vary greatly from site to site. For busi- 
ness sites, delays often incur real revenue loss, while 
for academic sites, the effects of delays are more 
intangible, resulting in missed grant deadlines, student 
attrition, and other “opportunities lost”. In the latter 
case, it is better to model cost as risk of potential loss 
rather than as tangible loss. 


We can best cope with uncertainty and risk by 
computing the expected value of each potential risk. 
Contingencies are like requests; they arrive during a 
period of vulnerability with a specific rate depending 
upon the severity of the vulnerability; these arrivals 
are often Poisson. The total expected value of an out- 
age or wait is the sum of expected incident costs, 
taken over the various kinds of incidents. If incidents 
arrive with a constant Poisson rate A, the expected 
incident cost is the number of expected incidents times 
the cost of an incident. This is in turn a product of the 
rate of arrival for the incident, the elapsed time, and 
the average cost per incident. Note that the word 
“incident” applies not only to security problems, but 
also to lost opportunities such as students dropping out 
of school, employees quitting, etc. 


Thus we can think of the cost function c,(¢) for a 
particular request 7 as 


19th Large Installation System Administration Conference (LISA ’05) 131 


Toward a Cost Model for System Administration 


c(t) = an) 1 Ci(t) (6) 
where c,,,(¢) represents tangible losses and c,,(t) repre- 
sents intangible losses. While c,,,(¢) represents work 
and revenue losses and is proportional to the scale of 
the outage, c,; represents possible losses due to ran- 
dom events. If contingencies are elements d of a set D, 
of all possible contingencies that can occur during 
request r, and contingencies in D, are statistically 
independent, then the cost c,; for all of them is the sum 
of their individual costs 

C,(0) 7 z Cyia(t) (7) 

deD, 
where c,;q is the contingency cost for deD, while 
waiting for r. If contingencies deD, have Poisson 
inter-arrival times Ay, then 


Cria = haCa (8) 
where C, is the average cost per incident for d. Thus 
c(t) oa Crm(0) + = haCu- (9) 
deD, 
If Cm Ags and Cy are constants, then 
a, =c,n+( ¥ AgCy) (10) 
deD, 


is also a constant, and 


fa 
C, = J etoae = 6,1, . (11) 

fy 
Note that there are a lot of if’s in the above justi- 
fication and the reader should be warned that assump- 
tions abound here. The formula for cost of waiting 
simplifies easily only if particular assumptions hold. 
As we make these assumptions, our model loses accu- 
racy and expressiveness. With all assumptions in 
place, we have Patterson’s model; as he states, it is an 

oversimplification. 


If requests can be divided into classes keK, each 
with a different proportionality constant o,, then the 
total cost of processing a set of requests is the total 
time spent waiting for each class, times the propor- 
tionality constant for that class. Thus, in the simplest 
case, the total cost of waiting is 

Cy = 2 Ey Ol, (12) 
keKreK 
or 
Cy= > oo St (13) 
kek reK . . 
Thus the contribution of each class & is proportional to 
the total time spent waiting for events of that class. 


In this approximation we make many simplifying 
assumptions: 
1. Contingencies arrive with Poisson rates. 
2. Contingencies are statistically independent of 
one another. 
3. The effect of a contingency does not change 
over time. 


These are limits on how we formulate a problem; 
we must not allow dependencies to creep into our clas- 
sifications. Part of this formulation is to think of bursts 
of events as single events with longer service times. 
For example, it is incorrect to characterize “bursty” 
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contingencies such as virus attacks as host events; 
these events are not independent of one another. How- 
ever, the event in which the virus entered the system is 
not bursty, independent of all other like events, and 
thus can be treated as one contingency. Likewise, spam 
from a particular site is not an independent event for 
each recipient, though spam from a particular source is 
often independent of spam from other sources. 


The main conclusion that we make from these 
observations is that 


The intangible cost of waiting for a request is, to 
a first approximation, proportional to time spent 
waiting (though the proportionality constant may 
vary by request or request class). 


While some constants remain unknown, the values for 
some proportionality constants are relatively obvious. 
If n users are affected by an outage, then the tangible 
cost of downtime is usually approximately propor- 
tional to 7. Likewise the rate of incidents that involve 
one member of the population (such as attrition) is 
usually approximately proportional to the number of 
people involved (due to independence of people as 
free agents). 


Estimating Service Rates 


In the above sections, we show a linear relation- 
ship between the cost of waiting and amount of time 
spent waiting, and show that the amount of time spent 
waiting depends upon arrival rate and service rate for 
tasks. In our observation of real systems, arrival rate 
was relatively easy to determine. To determine the 
cost, however, we must also know the service rate 
with which requests are completed. We cannot mea- 
sure this parameter directly; we can only measure the 
waiting time that results from it. How do we estimate 
the service rate itself? To answer this question, we 
borrow from a broad body of work on complexity esti- 
mation in software engineering [19]. 


Cost modeling in software engineering concerns 
the cost of maintaining a large piece of software (such 
as a program or configuration script). The basic strat- 
egy is to measure the complexity of the final product in 
some way, and then predict from that complexity how 
much it will cost to craft and maintain the program. 


Complexity metrics that can aid in determining 
cost of a software engineering project include both 
“white-box”” and “black-box” methods. A “black- 
box” method looks at the complexity of requirements, 
while a “‘white-box”” method looks at the complexity 
of a potential finished product. The goal of either kind 
of analysis is to produce a service rate that can be uti- 
lized for later analysis. To map this to system adminis- 
tration, a “white box” method would base cost esti- 
mates on the structure of practice, while a “black 
box” approach would base cost estimates upon the 
structure of the problem. 
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White-box Methods 


In software engineering, white-box software 
metrics include: 

1. Lines of code (LOC): the complexity of a soft- 
ware product is proportional to its length in 
lines of code. 

2. cyclomatic complexity [16]: the complexity of 
a piece of software is proportional to the num- 
ber of “if” statements in the code. 


It is generally agreed that cyclomatic complexity 
is a much better measure of complexity than LOC, for 
a variety of reasons, including variations in the expres- 
siveness of programming languages; long programs in 
one language can be short in another. The key is to 
find something about the program that is more closely 
related to its cost than its length. For programs, the 
number of branches contributes to the difficulty of 
debugging or maintaining the program. The key to 
white-box analysis of system administration is to find 
an analogue to the branches for programs. 















Procedure B 
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Whitebox analysis of programs inspires a similar 
form of analysis for system administration procedures. 
While white-box analysis of programs starts with 
pseudo-code, white-box analysis of practice starts with 
a recipe or instructions to service a particular kind of 
request. If we treat each recipe as a “program”, with 
“branches” at particular steps, then we can compute 
the average time taken for the recipe by keeping statis- 
tics on the number of times that individual steps are 
utilized in practice. This provides a way to come up 
with estimated rates for a procedure, given estimates 
for subparts of the procedure. 


Note that white-box analysis of a recipe for sys- 
tem administration is quite different than white-box 
analysis of a program. In the case of the program, the 
white-box measurement of complexity does not 
depend upon the input to the program. In system 
administration, the performance of a_ procedure 
depends upon the nature of the environment. A white- 
box estimate of the time needed to service a request is 






Figure 8: An example troubleshooting flowchart. 
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Figure 9: The flow graph corresponding to Figure 8. 
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a measure of both the complexity of the procedure and 
the complexity of the environment. 





mF1 mF2 mF3 
Figure 11: An annotated flow tree tracks statistics 
that can be used to compute average completion 
rate. 


One way of performing white-box analysis starts 
with an (acyclic) troubleshooting chart for a proce- 
dure. We start with a a troubleshooting chart (Figure 
8) that describes procedures to perform and branches 
to take after each procedure. We convert this to a flow 
graph (Figure 9) by representing only decision nodes. 
Since a typical troubleshooting chart has no loops, we 
convert this graph into a flow tree by duplicating 
nodes with two or more incoming edges (Figure 10). 
We then annotate branches in that tree with statistics 
to be collected or estimated about the branch (Figure 
11). These statistics allow us to compute the mean ser- 
vice rate for the whole tree. 


The key to the computation is that given that we 
know a service rate for the subtrees of a node in the 
tree, we can compute the service rate for the node 
itself. The nature of the computation is illustrated in 
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Figure 12. Suppose we know the service rate mB for 
subtree B and mC for subtree C. Suppose that we want 
to compute the service rate mA for A, and know for 
each branch out of A, the service rate for A given that 
it takes the branch (mAl,mA2) and the probability 
with which that branch is taken (pA1,pA2). If we take 
the branch from A to B, and A has service rate mA1, 
then the average service time for the branch is 
1/mA1 + 1/mB. If we take the branch from A to C, the 
average service time for the branch is 1/mA2 + I/mC. 
If we take the branch to B with probability pAl, and 
the branch to C with probability pA2, then the average 
service time for both is the expected value 





| 1 
A 1( —— + —) + pA2 +—). 14 
P COAl mB P nA? mC) 0) 


Thus the average rate is the reciprocal of this. 
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Figure 12: Computing average completion rate for a 
flow tree. 


To enable this computation, each edge in the pro- 
gram graph is labeled with two quantities: the mean 
service rate for the predecessor of the edge, given that 
this branch is chosen, as well as the probability that 
this branch is chosen. We can either measure these 
directly or estimate them by some method. One obvi- 
ous way to measure both rates and probabilities is to 
perform the procedure many times and record the 
number of times each node is visited, the average ser- 
vice time before taking each branch, and the number 
of times each branch is taken. Then the ratio of the 
times the branch is taken, divided by the times its par- 
ent is visited, is the sample probability that the branch 
will be taken. 


In this abstraction there hides an astonishing fact: 
the order in which decisions are made strongly affects 
the time-in-system for such a graph. While the rates 
are properties of the administrative process, the proba- 
bilities of branching are properties of the environment. 
Further, these probabilities are not conditional in the 
Bayesian sense; they are femporo-conditional in that 
they depend upon the previous occurrence of a spe- 
cific procedure. In Figure 11, the probability of going 
to B from A is not the conditional probability P(B|A), 
but the probability of B after A: the probability that we 
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choose B given that A has already been completed. 
Bayesian identities do not hold; any change in a proce- 
dure affects the sample probabilities of all branches 
after it in the script. 

One way to estimate branch probabilities in this 
model is that certain subtasks depend upon hetero- 
geneity that is routinely tracked. For example, one 
step might be to determine whether the affected host 
runs Solaris or Linux. In this case, the sample proba- 
bilities for the branches are both known in advance 
from inventory data. In the same way, one can esti- 
mate some branch probabilities from overall statistics 
on the sources of trouble within the network. 


Black-box Methods 


White-box methods depend upon the fact that the 
nature of practice is already known, i.e., we know the 
steps that people will take to accomplish tasks. In sys- 
tem administration, as in software, we are often asked 
to estimate the cost of a process without knowing the 
steps in advance. To do this, we must use a method 
that estimates cost based upon the complexity of the 
outcome rather than the complexity of the process. 


Black-box methods for measuring software com- 
plexity include COCOMO [2, 3, 4]: the complexity of 
software depends upon an overall characterization of 
the software’s character and mission. COCOMO 
depends upon use of one of two estimations of code 
complexity: 

1. “object points” [3, 4]: the complexity of a 
piece of software is proportional to the com- 
plexity of the objects it must manipulate. 

2. “function points”: the complexity of a piece of 
software is proportional to the number of func- 
tions that it must perform. 


The key idea in COCOMO is that there is a rela- 
tionship between the cost of maintaining a program 
and the complexity of its interactions with the outside 


world, though we may not know the exact nature of 


that relationship in advance. COCOMO is “tuned” 
for a specific site by correlating object or function 
points with actual costs of prior projects. COCOMO is 
site-specific; the relationship between complexity and 
cost varies from site to site. By computing a ratio esti- 
mating the relationship between requirements and 
capabilities, one estimates the time that will be taken 
to complete requirements. 


We can apply the idea of COCOMO to system 
administration in a very direct way. While the software 
model for function points considers open files, the anal- 
ogous concept of function points for a network service 
would describe that service’s dependencies and interre- 
lationships with others. We know that the number of 
dependencies within a service influences the cost of 
maintaining it; we do not know the exact relationship. 


For example, we might compute the function 
points for an apache web server by assigning a number 
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of points to its relationship with each of the following 
subsystems: the network (DHCP, DNS, Routing), the 
filesystem (protections, mounts), and the operating sys- 
tem (users and groups). In a function point model, each 
of these attributes is assigned a “weight” estimating 
how difficult it is for a system administrator to deal 
with that aspect of configuration and management. The 
sum of the weights is then an estimator of “how com- 
plex” the service will be to manage. 


The main difficulty with this approach is the num- 
ber of potential weights one must estimate; virtually 
every subsystem mentioned in the system administra- 
tor’s book of knowledge [11, 13] has to be assigned a 
weight. Further, these weights are not universal; they 
vary from site to site, though it is possible that similar 
sites can use similar weights. For example, weights 
assigned to subsystems vary greatly with the level of 
automation with which the subsystem is managed. 


The cost of providing any service depends not 
only upon the complexity of the service, but also upon 
the capabilities of the staff. Our next step in defining a 
function point estimate of the complexity of system 
administration is to derive a capability summary of the 
administrative staff and site in terms of service rate. 
Obviously, a more experienced staff deals with tasks 
more effectively than a less experienced one. Capabili- 
ties in the model might include end-user support, ser- 
vice support, architecture, etc. If each staff member is 
assessed and the appropriate attributes checked, and a 
sum is made of the results, one has a (rough) estimate 
of capabilities of one’s staff. This has similarities to 
the SAGE levels of system administrator expertise 
defined in the SAGE booklet on job descriptions [9]. 


The last step in defining a function point esti- 
mate of the complexity of system administration is to 
assess the capabilities maturity of the site itself. One 
might categorize the site into one of the following 
maturity categories [14]: 

1. ad-hoc: everything is done by hand. 

2. documented: everything is documented, no au- 
tomation. 

3. automated: one can rebuild clients instantly. 

4. federated: optimal use is made of network ser- 
vices. 


Again, each one of these has a weight in deter- 
mining the overall capabilities. The sum of admin- 
istrator capabilities and site capabilities is an estimate 
of overall “capability points”’ for the site. 

It can then be argued that the complexity of 
administering a specific subsystem can be estimated 
by a fraction 
estimated service points 


service rate = (15) 


estimated capability points 
where service points and capability points are sums of 
weighted data as described above. If the weights for 
capability points are rates in terms of (e.g.) service- 
points per hour, then the complexity is the average 
response time in hours to a request. 
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The overwhelming problem in tuning COCOMO 
for system administration is that tuning the model 
requires detailed data on actual measured rates. The tun- 
ing process requires regression to determine weights for 
each of the complexity and quality factors. This is accom- 
plished by studying a training set of projects with known 
outcomes and properties. To use COCOMO-like systems, 
we must be able to gather more accurate data on the rela- 
tive weights of subsystems than is available at present. 


Some Experiments 


So far, we have seen that we can estimate the 
cost of system administration via one of two models. 
“Black box” methods require that we assess the time 
impact of the complexities of the problem being 
solved, while “white box” methods require that we 
estimate the time taken for a detailed set of tasks. Of 
these methods, “black box’’ methods give us informa- 
tion more quickly, but these methods require that we 
“score” facets of the problem being solved as harder 
or easier than others. These scores must be developed 
via practice, but we can learn something about the rel- 
ative complexity of black-box attributes via simula- 
tion. By simulating practice, we can account for realis- 
tic human behaviors that cannot be analyzed via 
known queueing theory. We can also observe how real 
systems can potentially react to changes in a less ideal 
way than ideal queueing models suggest. Particularly, 
we can study behavior of queueing systems ‘on the 
edge”; almost out of control but still achieving a 
steady state. In our view, this situation describes more 
IT organizations than it should. 


The Simulator 


The simulator, written in C++, is basically an 
M/M/c queue simulator with the ability to simulate 
non-ideal (“‘non-product”’) behaviors. It assumes that 


900 
800 
700 
600 
500 
400 


300 


Cumulative time spent waiting 


200 


100 


0 100 200 300 





Couch, Wu, and Susanto 


we have c identical system administrators working 
24x7 and generates a variety of requests to which 
these ideal administrators must respond. One can vary 
the behavior of the system administrator and the 
request queue and measure the results. The input to 
the simulator is a set of classes of tasks, along with 
arrival and service rates for each task. The output is 
the time spent waiting for requests (by all users), both 
per time-unit of the simulation and overall. We assume 
for this simulator that the cost of waiting is a constant; 
a unit wait by a single individual results in some con- 
stant intangible cost. These simulator assumptions are 
very much less complex than real practice, but one can 
make some interesting conclusions from even so sim- 
ple a model. 


Diminishing Returns 


Our first simulation exercise is to study the 
effects of adding system administrators to a group ser- 
vicing simple requests. We set up a simple multi-class 
system with a varying number of system administra- 
tors all of whom have identical average response rates. 
There are four request classes, corresponding to 
requests whose service time is an average of 1, 3, 8, 
and 24 hours, respectively. The service rate of each 
request class is twice its arrival rate, creating a balance 
between arrivals and service. We ran the exact same 
simulation for two, three, and four system administra- 
tors. The cumulative time spent waiting for service is 
shown in Figure 13. There is clearly a law of dimin- 
ishing returns; the change in wait time from three to 
four system administrators does not significantly 
change the time spent waiting for service. 


Saturation 


Realistic system administration organizations can 
be faced with request volume that is impossible to 
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Figure 13: Diminishing returns when adding administrators to a queue. 
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resolve in the time available. We know from classical 
queueing theory that an M/M/c queuing system 
exhibits predictable response time only if A/cu < 1, 
where A is the arrival rate, c is the number of adminis- 
trators, and wl is the service rate per administrator. In 
other words, there has to be enough labor to go 
around; otherwise tickets come in faster than they can 
be resolved, the queue grows, and delays become 
longer and longer as time passes. 


Figure 14 shows the same simulation as before, 
but adds the case of one administrator. This seems like 
an unbalanced situation in which request rate is 
greater than service rate, but looking at waiting time 
per unit time (Figure 15) we see that waiting time is 
not always increasing with time. So although one 
administrator is very much slower than two or three, 
the situation is not completely out of control. Note, 
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however, that the situation of the single administrator 
is very sensitive to load; he is “on the brink of 
destruction.” Small changes in load can cause large 
variations in response time, and the cost of administra- 
tion due to waiting is “chaotic”, especially when a 
request when a long service time enters the queue. 
Nevertheless, on average, the time spent waiting 
varies directly with elapsed time of the simulation. 


Figure 16 shows incremental waiting time for a 
truly “saturated” system in which there is no way for 
administrators to ever catch up. We saturate the queue 
in the previous example by multiplying the arrival 
rates for all requests by four. In this case, one and two 
administrators are in trouble; queue length is growing 
linearly with time along with wait time. Figure 17 
shows the cumulative time for this example. In a satu- 
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Figure 14: One administrator performs very poorly compared to two, three, and four. 
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Figure 15: Incremental data for Figure 14 shows that utilizing one administrator leads to chaotic wait times. 
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increases linearly with elapsed time, the cumulative 
time spent waiting varies as the square of elapsed time. 


Brinksmanship 


We consider it a fair statement that many IT 
organizations run with A/cu quite close to 1. It is thus 
no surprise that it is very difficult for these organiza- 
tions to cope with changes that might increase load 
upon system administrators, even for a short time. 
There is a solution, but it is counter-intuitive. Figure 
18 shows the effect of a “catastrophic” flurry of 
requests arriving in a near-saturated system. For a 
short while, wait times go way up, because the system 
is already almost saturated and the new requests push 
it over the limit. The key is to distribute the same 
requests over a long time period (Figure 19), to avoid 
pushing the system over the limit and save waiting 
time. Note that in both figures, one administrator 
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alone simply cannot handle the load and chaotic waits 
occur in both cases. 


Lessons Learned 


Human systems self-organize around maximum 
efficiency for the task at hand, but not necessarily for 
future tasks. As such, we as system administrators are 
often near the “saturation point” in our practice. As in 
Figure 18, small increases in load can lead to cata- 
strophic delays. But the strategies in Figure 19 can help. 


One part of coping with being understaffed is to 
utilize automation to lessen workload, but that can lead 
to queue saturation in an unexpected way. The quandary 
of automation is that when something goes wrong, it is 
not one host that is affected, but potentially hundreds. If 
an automation mistake affects hundreds of nodes, we 
often have the situation in Figure 18; there are hundreds 
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Figure 16: Multiplying the arrival rate by four overloads one or two system administrators. 
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Figure 17: Cumulative wait time for overloaded system administrators varies as the square of elapsed time. 
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Voluntary Cooperation in 
Pervasive Computing Services 


Mark Burgess and Kyrre Begnum — Oslo University College 


ABSTRACT 


The advent of pervasive computing is moving us towards a new paradigm for computing in 
terms of ad hoc services. This carries with it a certain risk, from a security and management 
viewpoint. Users become increasingly responsible for their own hosts. A form of service 
transaction based on minimal trust is discussed. A proof of concept implementation of non- 
demand (voluntary) services is discussed for pervasive computing environments. ‘Voluntary 
Remote Procedure Call’ is a test-implementation of the proposed protocol integrated into cfengine, 
to show how voluntary cooperation of nodes can allow a cautious exchange of collaborative 
services, based on minimal trust. An analysis of implementation approaches followed by a 
discussion of the desirability of this technology is presented. 


Introduction 


Pervasive or ubiquitous computing is often wed- 
ded to the future vision of mobile and embedded 
devices, in smart homes and workplaces; however, per- 
vasive computing environments already exist today in 
Web Hotels and at Internet Service Providers. A chang- 
ing base of customers meets in an environment of close 
proximity and precarious trust and offers services. 


Pervasive mobile computing presents two inde- 
pendent challenges to content services. The first con- 
cerns how to secure platforms for virtual commerce. 
Here the problem is that one does not possess guaran- 
teeable credentials for those connecting to the ser- 
vices. The second concerns how to deal safely with 
anonymous, non-commercial services offered entirely 
on an ad hoc, cooperative basis. Here the problem is 
the risk involved in interacting at all with a client of 
unknown intentions. 


The former generally drives discussions about 
security and public key infrastructures (though this is 
of little help unless every user is independently veri- 
fied and identifies cannot be forged). The latter has 
made a name for itself through peer to peer file shar- 
ing such as Napster, Gnutella and other services. 
There is currently no technological solution that can 
determine the intentions of a client attempting to con- 
nect to a service. 


The challenges of pervasion are not only techni- 
cal, but also ‘political’. Autonomy is a noose by which 
to hang everything from security to system manage- 
ment in coming years. In the foreseeable future, one 
can expect pervasive services to be vying for attention 
on every street corner and in every building. The key 
ingredient that makes such a scenario difficult is “rust. 
Orderly and predictable behaviour is a precondition 
for technology that performs a service function, but if 
one connects together humans or devices freely in 
environments and communities that share common 


resources, there will always be conflicting interests. 
Anarchy of individual interests leads to contention and 
conflict amongst the members of the community; thus, 
as in the real world, an orderly policy of voluntary 
cooperation with norms is required that leads to suffi- 
cient concensus of cooperative behaviour. 


In this paper, we shall be especially interested in 
the administrative challenges of pervasion, such as 
services like backup, software updates, policy updates, 
directory services etc. We shall attempt to tackle a 
small but nevertheless important problem, namely 
how to ‘approach’ peers for whom one does not have 
automatic trust. 


In a pervasive setting, the the sheer multiplicity 
of devices and hosts demands autonomous administra- 
tion, for scalability. This means that decisions about 
trust cannot be reasonably processed by a human. 
Self-managing ‘smart’ devices will have to cope with 
decision-making challenges normally processed by 
humans, about the trustworthiness and reliability of 
unknown clients and servers. Devices must make 
political decisions, based on often fuzzy guidelines. 
This is not a problem that can be solved by public key 
infrastructures, since apparent certainty of identity is 
no guarantee for trust. 


Virtualization and the Multiplicity of Autonomous 
Agents 


In a few years, most services could well be pro- 
vided by virtual machines running on consolidated 
computing power [1, 2], leading to an even denser 
packing of competing clients and services. There are 
distinct advantages to such a scenario, both in terms of 
resource sharing and management. Users can then cre- 
ate and destroy virtual machines at will, in order to 
provide or consume services or manage their activities. 


The ability to spawn virtual clients, services and 
identities, quite autonomously, makes the interactions 
between entities in a pervasive environment full of 
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uncertainties. Rather than having a more or less fixed 
and predictable interface to the outside world, as we 
are used to today, users will become chameleons, 
changing to adapt to, or to defy, their neighbours’ 
wishes. What users choose to do with this freedom 
depends more on the attitudes of the individuals in the 
virtual society than on the technologies they use. 


Today, at the level of desktop computing, the 
trend from manufacturers is to give each user increas- 
ing autonomy of administrative control over their own 
workstations, with the option to install and configure 
software at whim. Thus each user can be either friend 
or foe to the community of behaviours — and the uni- 
formity of policy and configurations that many organi- 
zations strive for today will not be a certain scenario 
of the future. 


We are thus descending into an age of increasing 
autonomy and therefore increasing uncertainty about 
who stands for what in the virtual society. The aim of 
this paper is to offer a simple mechanism for host 
cooperation, without opening the hosts to potential 
abuses. It does this by allowing each individual agent 
to retain full control of its own resources. 


The plan for the paper is as follows: we begin 
with a discussion of the need for cautious service pro- 
vision in a pervasive environment, by using what is 
known about client-server relationships between 
humans. Some examples of management services are 
discussed that present a risk to both parties and it is 
suggested how a consentual, voluntary protocol might 
be implemented to reduce the risk to an acceptable 
level. A solution to negotiating voluntary cooperation 
is presented, which trades the traditional model of vul- 
nerable guaranteed service levels for increased secu- 
rity at the expense of less predictable service rates. An 
implementation of the protocol in the programming 
language Maude helps to enlighten the possible 
administrative obstacles which follows of the protocol. 
An example implementation in the system administra- 
tion agent framework cfengine is presented which 
incorporates voluntarism in the existing cfengine com- 
munication framework. A comparison and discussion 
of the two approaches concludes the paper. 


Client-server Model 


In the traditional model of network services, a 
client stub contacts a server on a possibly remote 
machine and requests a response. Server access con- 
trols determine whether the request will be honoured 
and server host load increases as a function of incoming 
requests. Clients wait synchronously for a reply. This is 
sometimes called a Remote Procedure Call (RPC). 

This model is ideal for efficiently expediting ser- 
vices that are low risk and take only a short time to 
complete, but it has a number of weaknesses for services 
that require significant processing or have basic admin- 
istrative consequences for the network: there is thus the 
risk of exploitation and denial of service vulnerabilities. 
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Management services are not typical of network 
services in general. By definition they are not supposed 
to be high volume, time critical operations such as web 
services or transaction processing services. They are 
nonetheless often of great importance to system secu- 
rity. In a pervasive computing environment, hosts are 
of many different types; many are mobile, partially 
connected systems under individual control, configured 
at the whim of their owner. A more defensive posture 
is therefore called for in the face of such uncertainty. 


This paper presents a simple protocol that 
assumes no automatic trust: a remote service model 
based entirely on ‘pull’ semantics, rather than ‘push’ 
semantics. Thus services can be carried out, entirely at 
the convenience of the answering parties: the risk, at 
each stage, is shifted to the party that is requesting 
communication. One interesting side-effect of this is 
that services become more resilient to Denial of Ser- 
vice attacks. 


Pervasion and Stable Policies 


The interactions between hosts can naturally lead 
to changes in their behaviour, e.g., if they exchange 
configurations, policy instructions or even software. In 
the ad hoc encounters that emerge between politically 
autonomous hosts, some stability must condense out of 
the disorder if one is to maintain something analogous 
to law and order in the virtual realm [3]. Law and 
order can only be built on cooperative concensus — it 
can only be enforced if a majority wants it to be 
enforced, and conforms in a scheme to enforce it. How 
such a concensus emerges is beyond the scope of the 
present paper, but will be discussed elsewhere [4, 5, 6]. 


Tourism 


In the present scenario, the model for contact 
between computers is like virtual tourism: visitors roam 
through an environment of changing rules and policies 
as guests, some of them offering and requesting ser- 
vices, like street sellers or travelling salesmen (no rela- 
tion to the famed algorithmical optimizer). The commu- 
nication in such encounters is de-centralized; it takes 
the form of a “peer-to-peer” relationship between par- 
ticipants. The snag with peer-to-peer transactions that it 
requires considerable trust between the participating 
hosts. Who is to declare that an arbitrary participant in 
the scheme is trustworthy [7]? 


It is known from game theoretical and evolution- 
ary simulations that stability in groups of collaborative 
agents is a precarious phenomenon, and thus caution in 
accepting arbitrary policy instructions is called for. If 
hosts alter their behaviour automatically in response to 
ad hoc interactions, then the effective behaviour of the 
group can easily be subverted by a rogue policy [8] and 
the illusion of organizational control can evaporate. 


The approach to collaboration which is adopted 
below is to assume the scenario of lawlessness men- 
tioned above, by placing the provision of administrative 
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services (e.g., backups, information or directory ser- 
vices, policy distribution etc.) on an entirely voluntary 
basis. This is, in a tangential way, a continuation of the 
idea of the “pull contra push” debate in system admin- 
istration [9, 10] (see Figure 1). 


Voluntary Cooperation 


There are three reasons why the traditional 
model of client-server communication is risky for per- 
vasive computing services (see Figure 1). 


Q? 


Figure 1: In a traditional client-server model, the 
client drives the communication transaction 
“pushing its request”. In voluntary RPC, each 
party controls its own resources, using only ‘pull’ 
methods. 


— 


. Contemporary network servers are assumed to 

be ‘available’. Clients and servers do not expect 
to wait significantly for processing of their 
requests. This impatient expectation leads to 
pressure on the provider to live up to the expec- 
tations of its clients. Such expectation is often 
formalized for long term relationships as Ser- 
vice Level Agreements (SLA). For ad hoc 
encounters, this can be bad both for the 
provider and for the client. Both parties wait 
synchronously for a service dialogue to com- 
plete, during which their resources can easily 
be tied up and exploited maliciously (so-called 
Denial of Service attacks). 
Non-malicious, random processes can also be 
risky. It is known from studies of Internet traf- 
fic that random arrival processes are often long 
tailed distributions, i.e., include huge fluctua- 
tions that can be problematical for a server [11]. 
Providers must therefore over-dimension ser- 
vice capacity to cope with seldom events, or 
accept being choked part of the time. The alter- 
native is to allow asynchronous processing, by 
a schedule that best serves the individuals in the 
transaction. 
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2. The traditional service provider does not have 
any control over the demand on its processing 
resources. The client-server architecture drives 
the servers resources at the behest of the client. 
This opens us to risk of direct attacks like 
Denial of Service attacks and load storms from 
multiple hosts. In other words, ad hoc encoun- 
ters do not have to trust only individuals but 
also the entire swarm of clients in a milieu col- 
lectively. The alternative is that hosts should be 
allowed to set their own limits. 

3. The traditional service provider receives requests 
that are accepted trustingly from the network. 
These might contain attacks such as buffer over- 
flows, or data content attacks like viruses. The 
alternative is for the server to agree only to check 
what the client is offering and download it if and 
when it can be handled safely. 


Although one often aims to provide on-demand, 
high-volume services where possible, it is probably 
not prudent to offer any service to just any client on 
demand, given how little one actually can know about 
them. Nor is it necessary for many tasks of a general 
administrative nature, where time is not of the 
essence, and a reply within minutes rather than sec- 
onds will do (e.g., backups, policy updates, software 
updates etc.). Avoiding messages from a client is a 
useful precaution: some clients might be infected with 
worms or have parasitic behaviour. 


The dilemma for a server is clearly that it must 
expose itself to risk in order to provide a service to 
arbitrary, short-term customers. In a long term rela- 
tionship, mutual trust is built on the threat of future 
reprisals, but in an opportunistic or transitory environ- 
ment, no such bond exists. Clients could expect to ‘get 
away’ with abusing a service knowing that they will 
be leaving the virtual country soon. 


In the voluntary cooperation model this risk is 
significantly reduced in the opening phase of estab- 
lishing relations: neither “client” nor “server” 
demand any resources of each other, nor can they 
force information on each other. All cooperation is 
entirely at the option of the collaborator, rather than at 
the behest of the client. Let us provide some examples 
of how voluntary collaboration might be used, 


Configuration Management in a Pervasive Setting 


A natural application for voluntary cooperation is 
the configuration or policy management of networked 
hosts. How shall roaming devices adapt to what is local 
policy [12]? A host that roves from one policy region to 
another might have to adapt its behaviour to gain 
acceptance in a scheme of local rules, just as a traveller 
must adapt to the customs of a local country. However, 
this does not imply automatic trust (see Figure 2). 
Today, many users think in terms of creating a Virtual 
Private Network (VPN) connection back to their home 
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base, but to get so far, they must first be allowed to 
interact with their local environment. 


Configuration management is a large topic that 
concerns the configuration of resources on networked 
hosts. See, for instance, [14] for a review. There are 
several studies examining how computing systems can 
be managed using mobile agents. However, mobile 
agents have not won wide acceptance amongst devel- 
opers [15] and they violate the risk criteria of the 
present paper. Opening one’s system to a mobile agent 
is a potentially dangerous matter, unless the agent fol- 
lows a methodology like the one described here. 


Three examples of autonomy in computer 
administration are shown in Figure 2. In (a) all 
instructions for configuration and behaviour originate 
from a central authority (a controller). This is like the 
view of management in the SNMP model [16, 17] and 
traditional telecom management models like TMN 
[18]. The dotted lines indicate that nodes in the net- 
work could transmit policy to one another to mitigate 
the central bottleneck; however, control rests with the 
central controller, in the final instance. In (b) there is a 
hierarchical arrangement of control [19, 20]. A central 
controller controls some clients individually, and 
offers a number of lesser authorities its advice, but 
these have the power to override the central controller. 
This allows the delegation of control to regional 
authorities; it amounts to a devolution of power. In (c) 
one has a completely decentralized (peer to peer) 
arrangement, in which there is no centre [2], 3]. 
Nodes can form cooperative coalitions with other 
nodes if they wish it, but no one is under any compul- 
sion to accept the advice of any other. 
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This sequence of pictures roughly traces the evo- 
lution of ideas and technologies for management. As 
technology for communication improves, management 
paradigms become naturally more decentralized. This 
is probably more than mere coincidence: humans fall 
naturally into groups of predictable sizes, according to 
anthropologists [22]. There is a natural rebellion 
against centralization once the necessary communica- 
tion freedoms are in place. Then groupings are essen- 
tially “buddy lists”, not central authorities. 


A decentralized scheme has advantages and dis- 
advantages. It allows greater freedom, but less pre- 
dictability of environment. Also, by not subscribing to 
a collaborative authority, one becomes responsible for 
one’s own safety and well-being. This could be more 
responsibility than some “libertarian” nodes bargain 
on. One could end up re-inventing well-known social 
structures in human-computer management. 


The immunity model of system administration 
describes how nodes can made to take responsibility 
for their own state, in order to avoid flawed external 
policies, bottle-necked resources and unnecessary 
assumptions about trust [10]. The trouble with making 
hosts completely independent is that they need to com- 
municate and even ‘out-source’ tasks to one another. 
The immunity model must therefore extend to com- 
munications between hosts that allow them to main- 
tain their autonomy, accepting and rejecting data as 
they please. Other suggestions for an RPC paradigm in 
a mobile environment include refs. [23, 24, 25]. 


Some examples applications for voluntary 
Remote Procedure Call (RPC) are presented below. 
Most of these address the impossibility of establishing 
true nature of a client in an ad hoc meeting. 
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Figure 2: Three forms of management between a number of networked host nodes. The lines show a logical net- 
work of communication between the nodes. Dark nodes can determine their own behaviour, 1.e., they have polit- 

ical freedom, while light nodes are controlled. The models display a progression from centralized control to 
complete de-centralization of policy (see [13, 3]). 
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1. Mobile hosts can use voluntary collaboration to 
check in to home base for services. For example, 
a laptop computer that requires a regular backup 
of its disk to home base, using a pull-only mech- 
anism will need to check in to home base 
because the server does not know its new IP 
address, or whether it is even up and running. 
(Using IPv6, a host could register its mobile 
address with home base to set up IPv6 forward- 
ing, but if a device has no home base, then this 
will not work.) The laptop would like to control 
when its resources are going to be used by the 
backup software and vice versa, rather than 
being suddenly subjected to a highly resource 
intensive procedure at an arbitrary time. 


With a voluntary agreement, the resulting 
backup time is a compromise between the 
clients wishes and the server’s wishes. It is not 
exactly predictable in time for either party, but 
it is predictable in its security. Hosts must 
learn the expected services times rather than 
assume them. 


(Again, it is emphasized that encryption cannot 
solve this problem. Merely opening an 
encrypted tunnel back to home base does not 
help the parties to manage their resources. It 
only helps them to establish their respective 
identities and to communicate privately.) 

2. Exchanges of security alerts and patches could be 
offered on a voluntary basis, e.g., warnings of 
recent port-scanning activity, detected anomalies 
or denial of service (including spam) attacks. 
Such warning could include IP addresses so that 
hosts could add the offending IP-port combina- 
tions to its firewall. Using this scheme, client 
hosts agree to signal one another about changes 
and patches which they have received. These 
could be cross referenced against trusted signa- 
ture sources downloaded separately — allowing 
an impromptu Trusted Third Party collaboration. 
This kind of technique is now being used for 
online operating system updates. 

3. The ultimate autonomy in distributed services 
is the free exchange of information and services 
between arbitrary hosts [12]. Voluntary accep- 
tance of policy or method data could be utilized 
as a ‘cautious flooding’ mechanism, for ‘viral 
spreading’ of data; e.g., in which each host sig- 
nals its neighbours about current policy or soft- 
ware updates it has received, allowing them a 
controlled window of opportunity to download 
the software (rather than simply opening its 
ports to all hosts at the same time). It is known 
that such mechanisms are often efficient at 
spreading data [26, 27] 

4. Neighbouring Service Providers with a stand- 
ing arrangement with one another could use the 
voluntary mechanism to sell each other server 
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capacity, using virtual machine technologies. 
When the demand on ISP1 becomes too high, a 
method SpawnVM() which is sent to the col- 
laborator ISP2. If ISP2 has the capacity to help, 
it spawns a virtual machine and returns the IP 
address. The method call could also include a 
specification of the system. 


These examples underline the need for individu- 
als to maintain control of the their own resources, but 
at the same time allow one another the potential to 
collaborate. 


Protocol 


There are two phases of voluntary cooperation: 
agreeing to work together (forming a contract) and 
fulfilling one’s obligations (if any) with minimal risk 
to oneself. Then, given an autonomous framework the 
implementation of a cautious service protocol is 
straightforward. 


The Negotiation Phase 


A willingness to cooperate between individuals 
must be established in any service before service pro- 
vision can be considered. This is one of the crucial 
phases and it brings us back to the opening remarks. 
Apart from the traditional client request, there are two 
ways to set up such an agreement: with or without 
money. One has the following options: 

1. A contract based on money: a client is willing 
to pay for a service and some form of negotia- 
tion and payment are processed, based on a 
legal framework of rules and regulations. This 
is a common model because it is rooted in 
existing law and can threaten reprisals against 
clients who abuse privileges by demanding a 
knowledge of their identity in the real world. 

2. No contract but registration: clients use the ser- 
vice freely provided they register. Again, they 
must surrender some personal details about 
themselves. This is a declaration of trust. A 
password or secret key can be used to cache the 
results once a real world identity has been 
established. Public key technologies are well 
acquainted with the problem of linking true 
identity to a secret key. We shall not discuss 
this further but assume the problem to be solu- 
ble. 

3. Open connection: the traditional way of con- 
necting to host services requires no registration 
or credentials other than an IP address that may 
or may not be used to limit access. 


In volume services, one usually wants to know in 
advance which clients to expect and what. their 
demands on service are: this expectation is guaranteed 
by access control rules, firewall solutions and service 
quality configuration. In voluntary cooperation models, 
service agreements cannot be made in the same way, 
since both parties have responsibilities to one another. 
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We do not consider the details of the negotiation 
process here, since it is potentially quite complicated; 
rather we shall assume, for this work, that a negotia- 
tion has taken place. The result of this process is a list 
of logical neighbours who are considered to be 
‘method peers’, i.e., peers who will execute remotely 
requested methods. 


Service Provision Phase 


After intentions have been established, one needs 
a way of safely exchanging data between contracted 
peers in such a way that neither party can subse- 
quently violate the voluntary trust model. To imple- 
ment this, we demand that a client cannot directly 
solicit a response from a server. Rather, it must adver- 
tise to trusted parties that it has a request and wait to 
see if a server will accept it of its own free choice. 

1. Host A: Advertises a service to be carried out, 
by placing it in a public place (e.g., virtual bul- 
letin board) or by broadcasting it. 

2. Host B: Scout A looks to see if host A has any 
jobs to do, and accepts job. Host B advertises 
the result when completed by putting the result 
in a public place. 

3. Host A: looks to see if host B has replied. 


Seen from the perspective of host A and B, this 
is done entirely using ‘pull’ methods, i.e., each host 
collects the data that is meant for it; no host can send 
information directly to another, thereby placing 
demands on the other’s resources. The algorithm is a 
form of batch processing by posting to local ‘bulletin 
boards’ or ‘blackboards’. There is still risk involved in 
the interaction, but each host has finer control over its 
own level of security. Several levels of access control 
are applied: 

¢ First the remote client must have permission to 
signal the other of its presence 

e Next the server must approves the request for 
service and connect to the client. 

e The client must then agree to accept the con- 
nection from the server (assuming that it is not 

a gratuitous request) and check whether it has 

solicited the request. 


The basic functionalities of this approach may be 
recognized as typical broker or marketplace approaches 
for agents in a multi agent system. This makes the discus- 
sion particularly interesting because it addresses a rela- 
tively known scenario. Before we go into into the admin- 
istrative challenges of this approach, we present one con- 
crete implementation as a protocol model simulation. 


Protocol Specification in Maude 


The programming language Maude [28] is spe- 
cializes in the simulation of distributed systems. In con- 
trast to typical programming approaches, this language 
is used to state the allowed transitions of a configura- 
tions state to another. This opens for searches in the 
execution domain of the entire system. With its roots in 
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transitional logic it has been used to analyze protocol 
related to mobile systems and security. The Maude 
framework enables the writer to focus on the essence in 
the protocol without wasting to much code on unrelated 
features. The process is especially helpful if one wants 
to examine prototype implementations of a protocol, for 
testing. A secondary effect is that formal specifications 
of a system often reveal hidden assumptions in the 
design of the protocol, which lead to differences in the 
protocol implementation and bugs. 


Programming in Maude consists mainly of two 
tasks: (1) defining the data model in terms of classes 
and types of messages between objects and (2) defin- 
ing the allowed transitions from one program state to 
another. It is thus declarative. Thus, instead of pro- 
gramming a particular path of execution, one sets the 
scene for all possible steps that might occur without 
assuming what happened before. Execution of a 
Maude specification is like rewriting logic. 


If one has a starting configuration and a number 
of independent rewriting rules, one could, in principle, 
use any of the possible rules. Maude creates a tree, 
where the initial configuration is the root node and all 
possible outcomes form the rest. For complex systems 
it is clear that the human mind can imagine only few 
of all the possible configurations. Maude possesses a 
search mechanism to actually traverse this tree to look 
for certain configurations. 


Maude takes an object-oriented approach when 
designing distributed systems. One usually defines 
‘class objects’ or data-types for the objects that are to 
communicate with each others, and a class for mes- 
sages. Below is an example of the classes specified in 
this implementation: A client (the node requesting ser- 
vice), a server, and a blackboard. 


class Client | pendingRPC : RPCList, 
knownServers OidList, 

jobs : JobList, 

completedJobs JobList, 

rpcEnabled OidList, 

blackBoard : Oid 
class Server | rpcClients : OidList, 
blackBoard : Oid 
class BB | rpcRequests : RPCList, 


rpcReplies RPCList 


To illustrate the specification of a transitional 
rule, consider the following: 


crl [accept-service-request] 
< S : Server | rpcClients : OL > 


(msg RPCrequestService(C,S) from C to §$ ) 


=> 
< § : Server | rpcClients : OL :: C > 


(msg RPCrequestReply(C.S,true) from S to C ) 


if ( not containsElement(OL, C) ) 


The “=>” marks the transition from the state 
before to the state after. The state before says only that 
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there is a Server object S and a message addressed to 
that object. The server object (enclosed in “< >’’) has a 
list of object pointers called rpcClients. The client is 
added to that list if it is going to be served by the server. 
Note that Maude assumes that there may be many other 
objects currently in the same state. The only thing that 
is specified is that, if at a given point a server S sees the 
message, it may react to it. Nothing is stated about time 
constraints or priorities of transitions. 


The displayed rule has a conditional that states 
that if a server sees a message addressed for it and con- 
taining a RPCrequestService(C,S) payload from client C 
it will answer with a reply but only if the client is not 
served by the server already. In this particular case it 
will answer “yes” to the request. A similar rule exists 
where the server answers “no”. Maude chooses one of 
these rules randomly in its simulation. 


Execution of a Maude program means supplying 
an initial state (also called Configuration) and letting 
Maude choose a path at random, based on the all the 
possible transitions of the given state. There is also the 
possibility of issuing searches in the execution tree (all 
possible paths) of the initial state in order to seek out 
undesirable states of the protocol. This is beneficial 
for system designers who work with distributed sys- 
tems too complex for unaided humans to think of, in 
every possible outcome. Other fields of computer sci- 
ence have used formal modelling of this kind for a 
long time. We perceive this tool to be of value in the 
field of theoretical system administration as well. 


Every execution tree is based on an initial configu- 
ration. To begin with something basic, lets take the case 
of one of client and one server. We define simplelnit to 
be a function that returns the following configuration: 


*** A simple initial state 
eq simpleInit = 
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he "BR" 
rpcRequests 


BB | rpcReplies : nil, 
nid > 
All participants know about each other out-of-band, and 
so they start with variables pointing to each other. The 
client has two jobs it needs to be performed, “‘job1” and 
“job2”. It has an empty list called completedJobs and 
the most basic search is to query whether this list can 
contain both jobs in the end. That would be a sign that 
the specification works as intended: 

(search [1] simpleInit =>+ 

< "client" Client 

completedJobs “Jobl” * “job2"., 

Client:AttributeSet > 

C’ :Configuration 


) 


The search gives the desired result, which indi- 
cates that the protocol can indeed be used. But the 
main goal of this implementation was to review any 
hidden assumptions in the design. After implementing 
the protocol in Maude, several unaddressed assump- 
tions became clear: 

e A Job is an atomic object that is translated into 
a single RPC call and does not depend on other 
Jobs or on a sequential execution of other jobs. 

e The service agreement request is a part of the 
actual RPC service and not a standalone ser- 
vice. 

e A client node knows the address of blackboard 
and server node beforehand. 

e A server node knows the address of the black- 
board. (The blackboard could be a local process 
on one of the participants to maintain complete 
autonomy, as in the cfengine implementation.) 

¢ Server and client node use the same blackboard 
in the simulation, but this sacrifices complete 
autonomy by introducing a third party. (This is 


< "client" : Client | completedJobs : empty, fixed in the cfengine implementation.) 
pendingRPC : nil, ¢ No explicit mention is made about the duration 
blackBoard : "BB", of the service agreement. The server will agree 
rpcEnabled : none, to serve the client indefinitely. 
knownServers server", ¢ A client has no way of ending a relationship 
jebe 2 "Joba" = "job" > witha server - 
< "server" : Server | blackBoard : "BB", ¢ We assume a trusted transmission medium. No 
rpcClients : none > lost or doubled messages can affect the result. 
4,5,12,13 BlackBoard 
1,2,3 





1. [send-service-request] 

2. [accept-service-request] 

3. [recieve-service-agreement] 

4. [send-RPC-request-to-bb] 

5. [recieve-RPC-request-from-client] 
6. [Server-send-pullJob] 





7. [ob-respond-to-pullJob] 
8. [bb-drops-pullJob] 
9. [server-does-job] 
10. [blackboard-saves-reply] 
12. [blackboard-returns-reply] 
13. [blackboard-drops-reply-no-finished-job] 


Figure 3: The different transitions in the Maude implementation of the protocol. The numbers represent the order of 


the transitions for a successful RPC call and reply. 
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Many of these assumptions are usually designed 
into RPC middleware, in order to provide trans- 
parency of these details. 


So, after this analysis, suppose we go back to the 
vision of wireless mobile devices that engage with each 
other using a protocol like the one described. What are 
the ramifications to the network and service level? 


Administrative Costs 


A RPC call will in the Maude implementation of 
the protocol requires least 8 message components to 
be sent if there are no redundant pull messages from 
either side. The frequency of pull messages from both 
client and server is not specified in the protocol itself. 
A smaller interval may saturate the network, and use 
more battery capacity in a mobile device, but gives 
better service. 


If we assume an equal transmission time ¢ for a 
message between each participant and the pull-interval 
of i and a time S for a server to process the job and 
that all other processing of messages is close to instan- 
taneous, we see that the smallest possible amount of 
time fora RPC call is: t*8+S . 


Also the expected number of messages can be 


§ 
described as: —+8 . 
i 


The term messages is used explicitly, because it 
may not correspond to the a single packet per mes- 
sage. It may be so for the pull and service request 
messages, since they would be small, but for the job 
and the result, there are no clear limits on how small 
or big they may be. One should also point out, that the 
messages, which constitute the “overhead” compared 
to a direct service, are also the smallest ones. The job 
and the result would have to be transmitted regardless 
of the protocol and should not be a measure of its effi- 
ciency. What may be noted, is that they travel twice in 
this protocol. So for a shared medium network, they 
will occupy the medium twice as long. 


Early in the text we pointed out the problem of 
current client-server technologies with regard to 
uncontrollable pressure on servers and the risk of bot- 
tlenecks and DoS-like saturations. Voluntary coopera- 
tion is proposed as a possible way of reducing this 
risk, although the negotiation phase of the protocol 
still contains direct traffic between the client and 
server. The less discussed participant in the plot, how- 
ever, is the blackboard. It will receive more traffic and 
data then any of the two nodes that actually are using 
the service. If one looks at the administrative ramifica- 
tions of this, one has in practice created a proxy which 
will get more network pressure than a typical client- 
server scenario would. Also, there is no voluntary 
cooperation in the blackboards part. Upon receiving a 
RPCjob it will have to store the RPCreply too at a 
later point. 
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A cfengine Implementation 


The voluntary RPC has been implemented as a 
cfengine collaborative application. Remote services 
are referred to one another as “‘methods” or function 
calls. An automated negotiation mechanism has not 
been implemented as this remains an open question. 
We begin by assuming that a negotiation has taken 
place that determines which hosts will be clients and 
which will be servers and what the nature of the ser- 
vice is. This establishes each hosts “Method Peers,” 
i.e., the hosts it is willing to provide services for. On 
the server host the update.conf rules must also contain 
a list of hosts to check for pending method requests. 

MethodPeers = ( clientl client2 ) 
MethodPeers = ( GetPeers(*) ) 


The client and server hosts must then have a copy of 
the invitation: 


methods: 


client_host||server_host:: 


MethodExample("Service request", parameter) 


action=cf.methodtest 
server=server_host 
returnclasses=ready 
returnvars=retval 


ifelapsed=120 
expireafter=/20 


alerts: 


MethodExample_ready:: 


"Service returned: $(MethodExample.retval)" 


The predicate classes in the line ending in double 
colons tells the software that this method stanza will 
be read both by the client and the server host. On the 
client it is a request, and on the server it is the invita- 
tion. The action and server attributes determine where 
the methods are defined and who should execute them. 
Return values and classes are declared serving as 
access control lists for returned parameters. Since a 
remote method is not under out local control, we want 
to have reasonable checks about the information being 
returned. Methods could easily be asked to return cer- 
tificates, for instance, to judge their authenticity. The 
ifelapsed and expireafter attributes apply additional 
scheduling policy constraints, as mentioned above. 

On server host a copy of the and invitation is 
needed to counter-sign the agreement. The module 
itself must then be coded on the server. The invitation 
is based on the naming of the hosts; this has caused 
some practical problems due to the trust model. The 
use of IP addresses and ties to domain names has 
proven somewhat unreliable due to the variety of 
implementations of Domain Name Service resolution 
software. A name might be resolved into an IPv6 
address on one end of a connection, but as an IPv4 
address on the other side. This leads to mismatches 
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that are somewhat annoying in the current changeover 
period, but which are not a weakness of the method in 
principle. See Display 1. 


A notable point regarding this implementation is 
that the protocol is incorporated into an existing ser- 
vice. This limits the range of possible applications, but 
reduces extra overhead since the RPC calls and results 
are bundled with the already scheduled communication. 


Incorporation of Voluntary RPC in cfengine 


The challenge of voluntary cooperation is to bal- 
ance a regularity of service (constraints on time) with 
the need for security against misuse attacks. The relia- 
bility of services becomes a matter for a host’s reputa- 
tion, rather than a matter of contract. Hosts must learn 
each others’ probable behaviour over time rather than 
demand a certain level of service. Clearly this is not 
acceptable in every situation, but it is a desirable way 
of opening relations with a host that one has never 
seen before. 


Given that all service execution is based on vol- 
untary cooperation, there are no guarantees about ser- 
vice levels. This is a feature of the protocol. However, 
this does not mean that arbitrary bounds on service 
levels are not implementable. Cooperative hosts can 
easily arrange for a rapid level of service by agreeing 
on a schedule for contacting their peers. This can be as 
often as each party feels is appropriate. 


If we assume that all host agents run diagnostics 
and administrative services with the same regular 


control: 

MethodName = ( MethodExample ) 
MethodParameters = ( valuel value2 ) 
MethodAccess = ( patterns ) 


## valuel is passed to this program, so lets 
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period P, and that all servers are willing to process the 
request without delay, then we can say that the 
expected time to service <7 is: 

<P) <<T) <2 «<P. 
This may be verified empirically. Limits can also be 
placed on the maximum number of times that a 
method can be evaluated in a certain interval of time. 


Note that the scheduling period P can be arbitrar- 
ily small, but for administrative activities one is usually 
talking about minutes rather than milliseconds. The 
efficiency of the method for small P is not impressive, 
due to the overheads of processing the additional steps. 
Thus the voluntary RPC is not suggested as a replace- 
ment for conventional service provision. 


In the cfengine software implementation, the 
locking parameters ‘ifelapsed’ and ‘expireafter’ deter- 
mine additional controls that bind the evaluation of 
remote methods tightly to policy. The ‘ifelapsed’ time 
says that a method will be ignored until the specified 
time has elapsed since the last execution. The 
‘expireafter’ option says that a still-executing method 
will not be allowed to continue for longer than the 
expiry time. 


Discussion and Comments 


The implementation in cfengine does not contain 
a negotiation phase as a part of the RPC protocol. As 
implemented, cfengine assumes that type of service and 
its parameters is established out-of-band. Also, there is 
no direct reference to any blackboard that caches the 
RPC interaction and thus there is complete autonomy. 


## add to it and send the result back for fun 


varl = ( "S{valuel}...and get it back" ) 
actionsequence = ( editfiles ) 
{HAHAHAHAHAHA AAR AAA AAA AAA HAE 
classes: 

moduleresult = ( any ) 


ready = ( any ) 


{HAREAHAAA AAA RAAAAAAAAAAAAAAAAAAAAAAAAAAAA 


editfiles: 
{ /tmp/filel 
AutoCreate 


AppendIfNoSuchLine "Important data...$(value2)" 


} 


{HERA BAAR AAA AAA AAA AAA AAA AAA 


alerts: 
moduleresult:: 


ReturnVariables("${varl}") 
ReturnClasses (ready) 


Display 1: Extended invitation example. 
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A more finely grained response regulation and 
access control is implemented in cfengine as com- 
pared to the more translucent concept of a “job” in 
the Maude implementation. There is little leeway for 
the nodes to change any of these parameters itself in 
any of the implementations and may therefore seem as 
not very dynamic or adaptable. It is also left to the 
implementation to sort out special cases, such as if the 
server would not want to serve the client but the RPC 
call is on the blackboard. Cfengine simply ignores 
such cases and clears them up as garbage. What coor- 
dination and messages would be necessary in order to 
remedy this situation? 


The implementation in cfengine raises an impor- 
tant question: if this were a truly pervasive environment 
and the nodes just got to know each other, how would 
they agree on the parameters and correct return type of 
the RPC? Why is this important? Because if the negoti- 
ation takes more computing power than the actual ser- 
vice in question then what is the gain for a mobile node 
to participate? Seen as a game, what is the cost-benefit 
relationship for the client and server? One has to think 
of the wider context. We must try to keep in mind 
future services that nodes may offer to each other, 
where such a negotiation is worth the delays. 


Based on our observations, it is natural to think 
of a realistic future scenario as being partially mobile, 
perhaps with some fixed infrastructure. that may serve 
as a cache for blackboard messages. For example, in a 
virtual shopping mall, a blackboard service might be 
provided, taking its own share of the risk from cus- 
tomers. Nodes have greater freedom through this pro- 
tocol: they may actually be away from the network 
while the contents are cached by the blackboard. A 
global ID, like the MIPv6 protocols home address, 
will make sure that the nodes can gather the data 
belonging to it from anywhere. It must be noted also, 
that the blackboard does not have to be a single hop 
away. It can be several, making this service more 
global and not just limited to a single network. 


So, is there really any need for this type of vol- 
untary cooperation? 


One model of pervasive services is commercial 
and based on the Internet café. Here roaming tourists 
pay for a ticket that grants them access to an otherwise 
closed local service. In regular commerce, it is 
assumed that this monetary relationship is sufficient to 
guarantee a decent form of behaviour, in which parties 
behave predictably and responsibly. But is this neces- 
sarily true? There are several faults in this assumption 
for network services. 


First of all, the commerce model assumes that 
virtual customers will behave much as they have done 
in human to human commerce. There is no compelling 
reason to suppose that this will be the case. A genera- 
tion of users has now grown up taking lawless online 
technologies for granted. Technologies such as mobile 
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phones, peer to peer sharing and Internet chat rooms 
are changing societal attitudes and the rules of interac- 
tion between young people [29]. 


Secondly, the model assumes that clients will 
exhibit predictable behaviour. Studies show that such a 
predictable relationship must be built up over time, 
and only applies if customers are regular patrons of a 
service provider [30]. Customer trust is based on the 
idea that customers and providers form relatively long 
lasting relationships and that one can therefore punish 
misdeeds by some kind of tit-for-tat interaction [8], 
because they will almost certainly meet again in the 
future. However, in a future where everything is on the 
move, distance has no meaning, and both children and 
adults are very much on-line, there is reason to sup- 
pose that clients will not ‘hit and run’ servers for prank 
or for malice. Hiding or changing identity on the Inter- 
net is trivial, so customers can visit providers repeat- 
edly with a new identity and there avoid reprisals. 


Finally, the use of of immediate payment as a 
barrier to deter frivolous usage assumes users will 
behave rationally in an adult fashion. This assumption 
is also flawed, since users might expect their uncoop- 
erative actions to have greater payoffs down the line. 
All of this points to the need for greater initial caution, 
and a reward scheme for cooperative behaviour. 


A simple protocol for voluntary cooperation has 
shown that with a slightly higher load on a network one 
can reduce the risks of connecting with untrusted 
clients. If these types of services arrive, using indepen- 
dent blackboards (e.g., Grid Computing), the system 
administrators may see the dawn of a new type of 
servers which act only as caches and proxies for volun- 
tary interaction. They will in turn be the ones that need 
protection from saturation and abuse. If fully 
autonomous behaviour is preserved, these problems can 
be avoided. We are currently working on a theory of 
fully autonomous interaction called promise theory [31]. 


Is voluntary cooperation a technical challenge or 
a human challenge then? The answer must be that it is 
both. A device that is misappropriated for malicious 
purposes by a human behaves simply as a_ hostile 
device to other computers. Computers perform a use- 
ful service to maintain the integrity of the networked 
community if they try to protect themselves, regard- 
less of the source of the mischief. 


Conclusions 


This paper discusses the issue of voluntary coop- 
eration in peer networks, and analyses a proposed 
trade-off which might help to lower the risks of client- 
server interactions in unpredictable environments. 


The mechanism proposed is one of minimal 
acceptable trust. Through the mechanism, hosts in a 
collective environment can maintain maximal adminis- 
trative control of their own resources, and hence reduce 
the risk of abuses and security breaches. Voluntary 
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cooperation (or pull-based service provision) is fully 
implementable and offers flexibility of asynchronous 
timing, as well as some additional security. 


The motivation for this work was initially to sup- 
port an administrative peer to peer algorithm that 
respects the autonomy of individual peers (inspired by 
model 6 of refs. [13, 3]), but the result is of more gen- 
eral interest. Its application domain is mainly tasks 
that are the traditional domain of the system admin- 
istrator, but which have to operate in an uncertain and 
far from controlled environment. 


A simulation in the analysis framework Maude, 
and an test implementation in the configuration tool 
cfengine [32, 33], show that the protocol functions 
properly, as advertised, and reveals the nature of the 
tradeoffs. The protocol overhead can be reduced, if 
voluntary cooperation is incorporated into an existing 
service, as in cfengine. 


We have tested the voluntary cooperation 
approach on mundane tasks like the backup of remote 
machines, distributed monitoring (exchange of usage 
data, confirmation of uptime etc.), load balancing and 
other tasks. Although this mechanism currently has a 
limited number of desirable applications, this is proba- 
bly due, in part, to the lack of truly mobile administra- 
tive approaches to hosts, in contemporary organiza- 
tions. As pervasive services mature, one could expect 
that a voluntary cooperation mechanism would actu- 
ally become invaluable for high risk transfers, e.g., in 
distributed software updating. 


For demand services, the minimal trust model 
seems both cynical and inappropriate: perhaps even a 
hindrance to cooperative behaviour and commerce. At 
some point, the actors of the virtual community online 
must learn the human rules of trusting engagement, 
and new generations of children will have to be edu- 
cated to accept that virtual communities are just as 
important as real ones. Human relations are based on 
rules of voluntary etiquette, but such social niceties 
have taken thousands of years to evolve. 


We present this work in the framework of system 
administration because system administrators are the 
virtual custodians of the hosts and devices in the per- 
vasive computing scenario. If administrators do not 
discuss the issues of scalable and management, and 
devise a workable scenario in the face of all its uncer- 
tainties, no one will. 
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ABSTRACT 


Complex, end-to-end network services are set up via the configuration method: each 
component has a finite number of configuration parameters each of which is set to a definite value. 
End-to-end network service requirements can be on connectivity, security, performance and fault- 
tolerance. However, there is a large conceptual gap between end-to-end requirements and detailed 
component configurations. To bridge this gap, a number of subsidiary requirements are created 
that constrain, for example, the protocols to be used, and the logical structures and associated 
policies to be set up at different protocol layers. 


By performing different types of reasoning with these requirements, different configuration 
tasks are accomplished. These include configuration synthesis, configuration error diagnosis, 
configuration error fixing, reconfiguration as requirements or components are added and deleted, 
and requirement verification. However, such reasoning is currently ad hoc. Network requirements 
are not even precisely specified hence automation of reasoning is impossible. This is a major 
reason for the high cost of network management and total cost of ownership. This paper shows 
how to formalize and automate such reasoning using a new logical system called Alloy. 


Alloy is based on the concept of model finding. Given a first-order logic formula and a 
domain of interpretation, Alloy tries to find whether the formula is satisfiable in that domain, i.e., 
whether it has a model. Alloy is used to build a Requirement Solver that takes as input a set of 
network components and requirements upon their configurations and determines component 
configurations satisfying those requirements. 


This Solver is used in different ways to accomplish the above reasoning tasks. The Solver is 
illustrated in depth by carrying out a variety of these tasks in the context of a realistic fault-tolerant 
virtual private network with remote access. Alloy uses modern satisfiability solvers that solve 
millions of constraints in millions of variables in seconds. However, poor requirements can easily 
nullify such speeds. The paper outlines approaches for writing efficient requirements. Finally, it 


outlines directions for future research. 


Introduction 


Complex, end-to-end network services are set up 
via the configuration method: each component has a 
finite number of configuration parameters each of 
which is set to a definite value. End-to-end network 
service requirements can be on connectivity, security, 
performance and fault-tolerance. However, there is a 
large conceptual gap between end-to-end requirements 
and detailed component configurations. To bridge this 
gap, a number of subsidiary requirements are created 
that constrain, for example, the protocols to be used, 
and the logical structures and associated policies to be 
set up at different protocol layers. 


By performing different types of reasoning with 
these requirements, different configuration tasks are 
accomplished. These include configuration synthesis, 
configuration error diagnosis, configuration error fix- 
ing, reconfiguration as requirements or components are 
added and deleted, and requirement verification. How- 
ever, such reasoning is currently ad hoc. Network 
requirements are not even precisely specified hence 


automation of reasoning is impossible. This is a major 
reason for the high cost of network management and 
total cost of ownership.’ This paper shows how to for- 
malize and automate such reasoning using the new 
concept of a Requirement Solver, as shown in Figure |. 


This Solver takes as input a set of network compo- 


nents and requirements upon their configurations and 


1“ operator error is the largest cause of failures...and 
largest contributor to time to repair..in two of the three 
(surveyed) ISPs .. configuration errors are the largest catego- 
ry of operator errors.” [1] 

“Although setup (of the trusted computing base) is much 
simpler than code, it is still complicated, it is usually done 
by less skilled people, and while code is written once, setup 
is different for every installation. So we should expect that 
it’s usually wrong, and many studies confirm this expecta- 
tion.” [2] 

“Consider this: ..the complexity (of computer systems) is 
growing beyond human ability to manage it..the overlapping 
connections, dependencies, and interacting applications call 
for administrative decision-making and responses faster than 
any human can deliver. Pinpointing root causes of failures 
becomes more difficult.” [3] 
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computes as output, component configurations satisfy- 
ing those requirements. The requirements can be in first- 
order logic (Boolean logic with quantifiers on individual 
variables). This logic is highly expressive and captures a 
very large class of practical network requirements. 


The Requirement Solver is used in different 
ways to accomplish the above reasoning tasks. The 
Solver is illustrated in depth by carrying out a variety 
of these tasks in the context of a realistic fault-tolerant 
virtual private network with remote access. The rea- 
soning tasks are accomplished as follows: 

1. Configuration Synthesis. To determine how to 
configure a set of components so they satisfy a 
system requirement R, submit the set of compo- 
nents and R to the solver and take the output. 

2. Requirement Strengthening. If a set of compo- 
nents satisfies a system requirement R but must 
now satisfy another requirement S, then to recon- 
figure components, submit the set of components 
and RAS to the solver and take the output. 

3. Component Addition. If a new component is 
to be added to a set of components already sat- 
isfying requirement R, then to configure the 
new component and possibly reconfigure exist- 
ing components, submit the new set of compo- 
nents and R to the Solver and take the output. 

4. Requirement Verification. To prove that it is 
impossible for an undesirable requirement U to 
be true when a set of components satisfies 
requirement R, submit the set of components 
and RAU to the Solver. If the Solver cannot 
find a solution, the assertion is proved. Other- 
wise, the Solver produces a counterexample. 

. Configuration Error Detection. To check 
whether configuration of a given set of compo- 
nents is consistent with a requirement R, repre- 
sent the configuration as a set C of constraints 
each of the form P=V where P is a configura- 
tion parameter and V its value. Then, submit 
the set of components and RAC to the solver. If 
the solver cannot produce a solution, a configu- 
ration error is detected. 

6. Configuration Error Fixing. If the configura- 

tion of a given set of components is inconsistent 


nN 


Set of Network Components 


Requirement 
Solver 
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with a requirement R, then submit the set of 
components and R to the Solver and find a new 
solution that is as “close” as possible to the 
current configuration. 


The Requirement Solver has been inspired by the 
new logical system called Alloy [4]. While Alloy is 
based in set theory, a subset of it also has an intuitive 
object-oriented interpretation: it lets one specify object 
types, their attributes and type of attribute values. It 
also lets one specify first-order logic constraints on 
these. Finally, it lets one specify a “scope” that 
defines a finite number of object instances of each 
type in a given system. 


Given a specification and a scope, Alloy 
attempts to find values of attributes of object instances 
in the scope that satisfy the specification. These values 
together constitute a “model” of the specification in 
the system in the logical sense of the word ‘*model”’. 
Alloy first compiles a specification into a proposi- 
tional formula in conjunctive normal form, then uses a 
satisfiability solver such as Berkmin [5] or Zchaff [6] 
to check whether the formula is satisfiable. If so, it 
converts satisfying values of propositional variables 
back into values of attributes and displays these. 


Often, more than one solution is found. Even 
though satisfiability is NP-complete in the worst case, 
in the average case, solutions are efficiently found. 
Modern satisfiability solvers can solve millions of con- 
straints in millions of variables in seconds. However, 
poor requirements can easily nullify such speeds. The 
paper outlines approaches for writing efficient require- 
ments. Finally, it outlines directions for future research. 


The Solver has a direct implementation in Alloy. 
Network component types, attributes and values are 
directly modeled in Alloy. A set of network compo- 
nents of different types is modeled as a scope. Net- 
work system configuration is modeled as values of all 
component attributes in a given scope. Network 
requirements are modeled using first-order logic. 
Solutions are found by the Alloy model finder. 


Subsequent sections illustrate Alloy’s capabilities, 
describe the design of a fault-tolerant VPN with remote 
access and the challenges of setting it up, outline a 


Network Requirements in First-Order Logic 
At and Across Multiple Protocol Layers 


Component Configurations Satisfying Requirements 
Figure 1: Requirement Solver. 
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formalization of the design in Alloy, describe how to 
accomplish, respectively, tasks 1-4 above, outline 
approaches for writing efficient requirements, outline 
relationship with previous work, summarize this work 
and the conclusions, and present directions for future 
research. 


Alloy By Example 


The three lines in Display | declare three object 
types: router, subnet and interface. The last type has 
two attributes, the chassis (of type router) to which it 
belongs, and network (of type subnet) on which it is 
placed. These attributes model configuration parame- 
ters of an interface. 


The predicate spec in Display 2 defines a specifi- 
cation whose model we will try to find. It is a conjunc- 
tion of three constraints. The first states that for every 
router X there is an interface y whose chassis is X, 1.e., 
every router has at least one interface. The second states 
that no two non-equal interfaces on the same router are 
placed on the same subnet. The third states that our net- 
work contains one router, two subnets and two inter- 
faces. These are the components we want to configure. 


The Alloy command shown in Display 3 pro- 
duces the model (values of configuration parameters) 
shown in Display 4. The first line states that the value 
of chassis for interface_0 is router_0 and the value of 
chassis for interface_1 is router_0. Similarly, for the 
second line. Alloy automatically creates instances of 
objects such as router_0, subnet_0, and subnet_1. Note 
that Alloy did not place interface_0 and interface_1 on 
the same subnet due to the second constraint. On the 
other hand, if we remove that constraint, Alloy pro- 
duces the additional solution show in Display 5 in 
which both interface_0 and interface_1 are placed on the 
same subnet. 


sig router {} 
sig subnet(} 
sig interface {chassis: router, network: 
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Fault-Tolerant Virtual Private Network With 
Remote Access 


Our top-level goal is to synthesize a fault-toler- 
ant network that enables hosts, including mobile hosts, 
at geographically distributed sites to securely collabo- 
rate. A network design for achieving this goal is now 
outlined. When this is implemented via configuration, 
one obtains a network of the type shown in Figure 2. 
The existence of a wide-area network, represented by 
the WAN router in the figure, is assumed. 


Each site has a gateway router called a spoke 
router whose external (or public) interface is connected 
to the WAN and whose internal (or private) interface is 
connected to hosts and servers in the site. A routing 
protocol is run on the external interfaces of spoke and 
WAN routers to automatically compute routes between 
these interfaces. As traffic between hosts and servers 
on different sites is intended to be private, it cannot be 
allowed to flow directly over the wide area network. In 
order to secure it, one possibility is to set up a full- 
mesh of IPSec tunnels between gateway routers. How- 
ever, full-mesh is not scalable since the number of tun- 
nels increases as the square of the number of sites: for 
the 200 sites expected in our domain, the number of 
tunnels would be nearly 20,000. 


A scalable alternative is a hub-and-spoke archi- 
tecture as shown. A certain number of hub routers is 
set up. Each spoke router sets up an IPSec tunnel to 
each hub router. Traffic from one site to another goes 
via two tunnel hops, one from its spoke router to a hub 
router and another from the hub router to the destina- 
tion site’s spoke router. The number of tunnels now 
only increases linearly with the number of sites. 


The problem, however, is that if a hub router fails, 
connectivity between sites is lost. This is because the 


subnet} 


Display 1: Declaring three object types. 


pred spec () 


{all x:router | some y:interface | y.chassis = x} 


{no disj xl,x2:interface | 


xl.chassis=x2.chassis && xl.network = x2.network} 
(#router=1 && #subnet=2 && #interface=2} 


Display 2: A specification with three constraints. 


run spec for 1 router, 2 subnet, 2 interface 


Display 3: Alloy command to create a model. 


chassis {interface_O -> router_0O, interface_l -> router_0} 
network : = {interface_0 -> subnet_l, interface_l -> subnet_0} 
Display 4: Results of command in Display 3. 
chassis : = {interface_0 -> router_0O, interface_l -> router_0} 
network : = {interface_0 -> subnet_0, interface_1 -> subnet_0} 


Display 5: Less constrained solution. 
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source spoke router will continue to send _ traffic 
through the IPSec tunnel to the failed hub router. 
IPSec has no notion of fault-tolerance that will enable 
the source spoke router to redirect traffic via another 
hub router. 


Routing protocols such as RIP or OSPF accom- 
plish precisely this kind of fault-tolerance, however 
they are incompatible with IPSec. They do not recog- 
nize an IPSec tunnel as directly connecting two routers 
since there can be multiple physical hops in between. 
The solution is to set up a new type of tunnel, called 
GRE, between each hub and spoke router. The pur- 
pose of GRE is to create the illusion to a routing pro- 
tocol that two routers are directly connected, even 
when there are multiple physical hops in between. 


This is done by creating new GRE interfaces at 
each tunnel end point and making these belong to the 
same point-to-point subnet. Now, if a hub router fails, a 
routing protocol will automatically direct traffic 
through another GRE tunnel to another hub router, and 
then to the destination. Each GRE tunnel is then 
secured via an IPSec tunnel. Thus, the required fault- 
tolerant virtual private network is set up. If two hub 
router failures are to be tolerated, then three hub routers 
are required. Then, the number of tunnels to be set up 
is just 600 (number of hub routers x number of spoke 
routers) or 3% of nearly 20,000 in the full mesh case. 


This solution has a useful defense-in-depth fea- 
ture: there are two separate routing domains, the exter- 
nal one and the overlay one. No routes are exchanged 
between these. Thus, even if an adversary compromises 
the WAN router, he cannot send a packet to a host. The 
WAN router does not even have a route to the host. 


In order to enable remote users to securely col- 
laborate, a remote access server is set up in “parallel” 
with a spoke router. A remote user connected to the 
WAN sets up an L2TP tunnel between his host and the 


IPSec Tunnel e@ Hub Router « 
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GRE Tunnel 
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server. This tunnel gives the illusion to the host of 
being directly connected to the internal interface of the 
sites’ spoke router. Consequently, all traffic between 
the host and any other host or server on the VPN is 
also secured. Again, one has to ensure that two sepa- 
rate routing protocols run on the access server, one for 
the private side and one for the public. In order to real- 
ize the above design, the following types of configura- 
tion parameters need to be set: 

1. Addressing: Router interface address, type and 
subnet mask. 

2. IPSec: Tunnel end points, hash and encryption 
algorithms, tunnel modes, preshared keys and 
traffic filters. 

3. OSPF: Whether it is enabled at an interface, 
and OSPF area and type identifiers. 

4. GRE: Tunnel end points and physical end 
points supporting GRE tunnels. 

5. Firewall: Policies at each site. 

6. Remote access: Subnets to which remote access 
server interfaces belong and routing protocols 
enabled on these. 


It is very hard to compute values of the above 
configuration parameters. The types of configuration 
errors that can arise are: 

1. Duplicate IP addresses may be set up, or all inter- 
faces on a subnet may not have the same type. 

2. IPSec tunnels may be set up incorrectly. For 
example, the preshared key, hash algorithm, 
encryption algorithm, or authentication mode 
may be unequal at the two tunnel end points. Peer 
values may not be mirror images of each other. 
These errors can lead to loss of connectivity. If 
the wrong traffic filter is used, then sensitive data 
can be transmitted without being encrypted. 

3. OSPF routing domain may be set up incorrectly, 
for example, it may not be enabled at a required 
interface or the area and type identifiers may be 


All internal interfaces and GRE 
tunnels such as these are in 
RIP domain 
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Figure 2: Fault-tolerant virtual private network with remote access. 
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incorrect. This can lead to incorrect routing 
tables and to outright isolation of subnets. 

4. Routing loops may arise. If the same OSPF 
process is also used for routing between the 
gateway and WAN routers, then if it does not 
find a path through the physical network it will 
attempt to find a path through the overlay net- 
work. Since the overlay network is supported 
by the physical network, a routing loop will 
arise. This problem can be mitigated by using 
two distinct routing protocols, one for the over- 
lay and another for the WAN. 

5. GRE tunnels may be set up incorrectly. For 
example, the peer values may not be mirror 
images of each other, or the mapping between 
GRE ports and physical ports may be incorrect. 

6. Firewall policies may block IPSec traffic, hence 
no traffic will pass through the tunnels. 

7. Remote access interfaces may not belong to the 
correct subnets and incorrect routing protocols 
may be configured on these. 


Before we show how to formalize the above 
design in Alloy, we capture its main intuitions in the 
following requirements: 

¢ RouterInterfaceRequirements 

1. Each spoke router has internal and external 

interfaces 

2. Each access server has internal and exter- 

nal interfaces 

3. Each hub router has only external interfaces 

4. Each WAN router has only external inter- 

faces 
¢ SubnettingRequirements 

5. A router does not have more than one 

interface on a subnet 

6. All internal interfaces are on internal sub- 

nets 

7. All external interfaces are on external sub- 

nets 

8. Every hub and spoke router is connected to 

a WAN router 
9. No two non-WAN routers share a subnet 
¢ RoutingRequirements 
10. RIP is enabled on all internal interfaces 
11. OSPF is enabled on all external interfaces 
¢ GRERequirements 
12. There is a GRE tunnel between each hub 
and spoke router 
13. RIP is enabled on all GRE interfaces 
¢ SecureGRERequirements 
14. For every GRE tunnel there is an IPSec 
tunnel between associated physical inter- 
faces that secures all GRE traffic 
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e AccessServerRequirements 
15. There exists an access server and spoke 
router such that the server is attached in 
“parallel” to the router 
¢ AccessControlPolicyRequirements 
16. Each hub and spoke external interface per- 
mits esp and ike packets 


The interesting fact is these requirements do not 
specify the number of sites in the VPN. Rather, they 
apply to all sites. As new sites are added, these 
requirements are instantiated for the extended network 
to determine how to configure new components and 
reconfigure existing ones. This is a hard problem, in 
general, but that we show how to automatically solve 
with our approach. 


Requirement Formalization In Alloy 


This section presents an Alloy formalization of net- 
work component types, subtypes and their attributes. It 
also presents a formalization of Requirements 12 and 14. 
The complete formalization is available in the longer 
report at http://alloy.mit.edu/papers/NetConfigA lloy.pdf. 
Various types of routers are modeled using the following 
Alloy type declarations (signatures): 

sig router {} 
sig wanRouter extends router {} 
sig hubRouter extends router (} 
sig spokeRouter extends router {} 
sig accessServer extends router {} 
sig legacyRouter extends router (} 
A generic interface just has a single attribute, the rout- 
ing protocol enabled at it. 


sig interface {routing:routingDomain} 


A physical interface has two attributes, the router 
chassis on which it is mounted, and the network on 
which it is placed: 

sig physicalInterface extends interface 


chassis: router, 
network: subnet} 


There are internal and external interfaces. Exter- 
nal interfaces are of two types, one on hubs and one 
on spokes; see Display 6. There are two types of rout- 
ing domains, RIP and OSPF: 

sig routingDomain (} 

sig ripDomain extends routingDomain {} 

sig ospfDomain extends routingDomain {} 
There are two types of subnets, internal and external: 

sig subnet{} 

sig internalSubnet extends subnet{} 

sig externalSubnet extends subnet! } 


There are three types of protocols, IKE, ESP and GRE: 


sig internalInterface extends physicallInterface (} 

sig externalInterface extends physicalInterface {} 

sig hubExternalInterface extends externalInterface {} 
sig spokeExternalInterface extends externalInterface {} 


Display 6: Interfaces on hubs and spokes. 
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sig protocol {} 

sig ike extends protocol (} 
sig esp extends protocol {} 
sig gre extends protocol {} 


A firewall policy contains one of two possible permis- 
sions, permit and deny, that respectively, mean 
whether the firewall should allow a packet to go 
through or be dropped. 

sig permission {} 


sig permit extends permission (} 
sig deny extends permission {} 


A firewall policy, shown in Display 7, defines whether 
a packet associated with a protocol is allowed to go 
through or dropped, as it leaves an interface. An IPSec 
tunnel encrypts all packets associated with a protocol 
entering at either its local or its remote endpoint. 
sig ipsecTunnel { 
local: externalInterface, 
remote: externalInterface, 
protocolToSecure: protocol} 


A GRE tunnel encapsulates a packet into a new packet 
with source address that of its local endpoint and des- 
tination address that of its remote endpoint. Also, the 
tunnel is considered a proper link in a routing domain. 
sig greTunnel { 
localPhysical: externalInterface, 
routing: routingDomain, 
remotePhysical: externalInterface} 
An IP packet’s attributes are its source and destination 
interfaces and the protocol it embodies. The precise 
data it carries is not modeled, since it is not relevant 
for our design purposes. 
sig ipPacket { 
source:interface, 
destination:interface, 
prot:protocol} 


Display 8 shows the Alloy version of Requirement 12. 
This states that between every hubExternallnterface x and 
spokeExternallnterface y there is a greTunnel whose local 
physical is x and remotePhysical is y, or vice versa. 


sig FirewallPolicy { 
prot: protocol, 
action: permission, 
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Display 9 shows the Alloy version of Require- 
ment 14. This states that for every greTunnel g there is 
an ipsecTunnel p that secures the gre protocol and 
whose endpoints are the same as the physical end- 
points of g. 


Configuration Synthesis 


This section shows how to synthesize the initial 
network with connectivity and routing. Define: 
PhysicalSpec = 
RouterInterfaceRequirements a 
SubnettingRequirements a 
RoutingRequirements 
In Alloy, this would be expressed as: 
Pred PhysicalSpec () { 
RouterInterfaceRequirements () 


SubnettingRequirements () 
RoutingRequirements ()} 


Define a scope consisting of 1 hubRouter, 1 spokeRouter, 
1 wanRouter, 1 internalinterface, 4 externallnterface, 1 
hubExternallnterface, 1 spokeExternallnterface, 1 ripDomain, 
1 ospfDomain, 3 subnet, 0 legacyRouter. These are the 
objects we want to configure. Now request Alloy to 
find a model for PhysicalSpec in the above scope. It 
synthesizes the network shown in Figure 3. It does so 
by producing the values of configuration parameters 
shown in Display 10. These are just the textual version 
of the network in Figure 3. Also note that spoke and 
hub routers are not directly connected, in accordance 
with Requirement 9. 


Requirement Strengthening 


In order to add an overlay network to the previ- 
ous one, extend the previous scope with a GRE tunnel 
then request Alloy to satisfy (PhysicalSpec ~ 
GRERequirements). Alloy synthesizes the network 
shown in Figure 4a. Alloy automatically sets up the 
GRE tunnel between the spoke and hub router and 
enables RIP routing on the GRE tunnel. 


To make GRE tunnels secure, extend the previous 
scope with an IPSec tunnel and request Alloy to satisfy 


protectedInterface: physicalInterface} 


Display 7: Firewall policy. 


{all x:hubExternalInterface, y:spokeExternalInterface | some 


g:greTunnel | 


(g.localPhysical=x && g.remotePhysical=y) or 
(g.localPhysical=y && g.remotePhysical=x) } 


Display 8: Requirement 12 in Alloy. 


{all g:greTunnel | 


some p:ipsecTunnel | p.protocolToSecure=gre && 


((p.local = g.localPhysical && p.remote = 


g-.remotePhysical) or 


(p.local = g.localPhysical && p.remote = g.remotePhysical)) } 


Display 9: Requirement 14 in Alloy. 
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(PhysicalSpec ~ GRERequirements ~ SecureGRERequire- 
ments). Alloy synthesizes the network in Figure 4b. 
Alloy automatically places the IPSec tunnel between the 
correct physical interfaces to protect the GRE tunnel. 


In order to add an access server to this network 
extend the previous scope with an access server, one 
internal interface, and one external interface and 
request Alloy to satisfy (PhysicalSpec ~ GRERequire- 
ments ~ SecureGRERequirements ~ AccessServerRequire- 
ments). Alloy synthesizes the network in Figure 4c. 
Note that the access server is placed in parallel with 
only the spoke router, not with any other router, and has 
the correct routing protocols enabled on its interfaces. 
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Component Addition 


When new components are added to an infrastruc- 
ture, the logic that governs infrastructure has to be 
instantiated to the extended set of components. This is a 
nontrivial problem for humans to cope with. With Alloy, 
this instantiation is accomplished simply by finding a 
model of requirements for the existing scope extended 
by new components. (In order to avoid reconfiguring 
existing components, one can strengthen the require- 
ment with existing configurations, each modeled as an 
equality P=V, P a configuration parameter and V its 
value). For example, in order to add a new spoke site to 


the previous network, extend its scope with a spoke 


routing = = 
{externalInterface_0 -> ospfDomain_O, 
externalInterface_1 -> ospfDomain_O, 
hubExternalInterface_0 -> ospfDomain_O, 
internalInterface_O -> ripDomain_0O, 
spokeExternalInterface_0 -> ospfDomain_0} 
chassis 7 = 
{externalInterface_0 -> wanRouter_0O, 
externalInterface_l -> wanRouter_0O, 
hubExternalInterface_0 -> hubRouter_0O, 
internalInterface_0 -> spokeRouter_0O, 
spokeExternalInterface_0 -> spokeRouter_0} 
network : = 
{externalInterface_O -> externalSubnet_l, 
externalInterface_1 -> externalSubnet_0O, 
hubExternalInterface_0 -> externalSubnet_0O, 
internalInterface_0 -> internalSubnet_0O, 
spokeExternalInterface_0 -> externalSubnet_1} 


Display 10: Configuration parameters for Figure 3. 


Hub 
RIP Domain Router 


OSPF Domain 





Spoke -—_—_—_—_@--- WAN 
Router Router 


Requirement Solver generates 
solution. Note that Hub and Spoke routers 
are not directly connected, due to Requirement 9 


Figure 3: Configuration synthesis: Physical network. 


(PhysicalSpec = RouterInterfaceRequirements a 
SubnettingRequirements a RoutingRequirements) 
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Figure 4a: Requirement strengthening: Adding overlay. 
(PhysicalSpec ~ GRERequirements). 
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Figure 4b: Requirement strengthening: Securing overlay. 
(PhysicalSpec a GRERequirements a SecureGRERequirements) 
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Figure 4c: Requirement strengthening: Adding remote access server. 
(PhysicalSpec a GRERequirements a SecureGRERequirements ~ AccessServerRequirements) 
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router, one internal subnet, one external subnet, one 
GRE tunnel and one IPSec tunnel. Requesting Alloy to 
synthesize a network satisfying (PhysicalSpec ~ GRE- 
Requirements ~ SecureGRERequirements ~ AccessServer 
Requirements) in the new scope yields the network in 
Figure 5a. 


Note that the new spoke router is physically con- 
nected just to the WAN router as required by Require- 
ment 8. Moreover, GRE and IPSec tunnels are auto- 
matically set up between the new spoke router and hub 
router and physical interfaces and GRE tunnels are 
placed in the correct routing domains. 


In order to add a new hub site to this network, 
extend its scope with a hub router, one external inter- 
face, one external subnet, two GRE tunnels and two 
IPSec tunnels. Requesting Alloy to synthesize a network 
satisfying (PhysicalSpec 1 GRERequirements ~ SecureGRE 
Requirements ~ AccessServerRequirements) in the new 
scope yields the network in Figure 5b. 


Finally, in order to permit IKE and ESP (proto- 
cols of IPSec) packets through the physical interfaces 
of hub and spoke routers, one can extend the above 
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scope with eight firewall policies, then request Alloy to 
satisfy FullVPNSpec = (PhysicalSpec ~ GRERequirements 
A SecureGRERequirements ~ AccessServerRequirements 
A FirewallPolicyRequirements). Alloy then synthesizes 
the network of Figure 2 without the hosts. The reason 
for 8 firewall policies is that one policy is required for 
each IPSec tunnel endpoint. 


Requirement Verification 


Identifying Incorrect Firewall Policies 


When we deployed the above network, we were 
careful to allow IKE and ESP packets to be permitted 
by access control lists at physical interfaces of hub and 
spoke routers. This was the reason for FirewallPoli- 
cyRequirements. However, we discovered that end-to- 
end connectivity was still not established. After consid- 
erable testing and analysis we realized that the WAN 
router itself was blocking IKE and ESP packets. We had 
not anticipated this cause. We now show how to formal- 
ize identification of this cause. 


Alloy was not used for such identification, but this 
example illustrates how it could have been. Define a 


e HubRouter 


e 
Spoke &> 5 Spoke ‘ <> 
Router WAN Router 


Router 





Figure 5a: Component addition: Adding new spoke router. 
(PhysicalSpec ~ GRERequirements a SecureGRERequirements a AccessServerRequirements) 
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Figure 5b: Component addition: Adding new hub router. 


(PhysicalSpec ~ GRERequirements a SecureGRERequirements a AccessServerRequirements) 
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condition called BlockedIPSec capturing conditions under 
which an IPSec packet can be blocked, and find out how 
it is possible that (FullVPNSpec ~ BlockedIPSec) be true. 
In other words, is it possible that the network be config- 
ured in a manner consistent with FullVPNSpec yet block 
IPSec packets? If so, we would have to modify require- 
ments to preclude this possibility. The predicate in Dis- 
play 11 states that IPSec is blocked if there is some esp 
or ike packet which is blocked. The predicate in Display 
12 states that a packet is blocked if there is some fire- 
wall policy protecting an external interface that denies 
the protocol for that packet. 


If we increase the scope of the last network to 
include 9 (more than 8) firewall policies, and request 
Alloy to find a model for (FullVPNSpec « BlockedIPSec), 
Alloy produces the values in Display 13 for prot, per- 
mission and protectedInterface attributes of firewall 
policies. In other words, firewallPolicy_8, applied on exter- 
nallnterface_O0 on the WAN router, blocks ike_0. 


Identifying Private Subnet Advertisement Of Pri- 
vate Subnets Into WAN 


This section illustrates another problem that arose 
during deployment of our VPN solution into an exist- 
ing network. Existing networks contain “legacy” 
routers that have no concept of internal or external sub- 
nets as do spoke routers. Thus, if our VPN is grafted 
into an existing network as shown in the figure above, 
the defense-in-depth feature mentioned in the second 
section is compromised. The legacy router can run the 


pred BlockedIPSec () { © 
some p:ipPacket, s,t:externalInterface | 
p.source = 


s && p.destination 
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same routing protocol on both its internal and external 
interfaces and thereby export the internal subnet Y to 
the WAN. Now, if the WAN router is compromised, an 
adversary can send packets to the host at Y. We now 
show how to formalize identification of this possibility. 


We define the code shown in Display 14. This pred- 
icate states that an internal subnet is advertised to the 
WAN if there is a legacy router with two interfaces, one 
attached to an internal subnet and another to an external 
subnet, and both have the same routing protocol enabled 
on them. Now, if we increase the scope of the network of 
Figure 5b to include one additional (legacy) router and 
two additional physical interfaces, and request Alloy to 
find a model for (FullVPNSpec ~ intemalSubnetAdvertised- 
ToWan), Alloy produces the the code in Display 15. In 
other words, physicallnterface_O and physicallnterface_1 can 
be placed on legacyRouter_0, one can be connected to an 
internal subnet, the other to an external subnet, and yet 
both can belong to ospfDomain_0. 


Writing Efficient Requirements 

Scope Splitting 

One critical parameter to control in Alloy is the 
size of the scope. If it gets too large it should be split 
up and the specification changed, if necessary. Con- 
sider the following specification declaring router and 
interface types, and a relation chassis mapping an inter- 
face to its router. Also define EmptyCond to be an 
empty set of constraints to satisfy (Alloy requires 
some constraint before it can be run): 


t && (p.prot = 


ike or p.prot=esp) && Blocked (p) } 


Display 11: Blocking IPSec if esp or ike packet is blocked. 


pred Blocked(pack:ipPacket) { 


some p:firewallPolicy, x:externalInterface | 


p.protectedInterface = x && 
p.prot=pack.prot && 
p.action = deny 

} 


Display 12: Block a packet if firewall policy denies its protocol. 








prot, : = firewallPolicy_4 -> permit_0O, 
{firewallPolicy_0O -> ike_O, firewallPolicy_5 -> permit_O, 
firewallPolicy_1l -> ike_O, firewallPolicy_6 -> permit_0O, 
firewallPolicy_2 -> ike_O, firewallPolicy_7 -> permit_0O. 
firewallPolicy_3 -> ike_O, firewallPolicy_8 -> deny_0} 
firewallPolicy_4 -> esp_0, protectedInterface = 
firewallPolicy_5 -> esp_O, {(firewallPolicy_0O -> spokeExternalInterface_l, 
firewallPolicy_6 -> esp_O, firewallPolicy_l -> spokeExternalInterface_0O, 
firewallPolicy_7 -> esp_O, firewallPolicy_2 -> hubExternalInterface_l, 
firewallPolicy_8 -> ike_0} firewallPolicy_3 -> hubExternalInterface_O, 

permission: = firewallPolicy_4 -> spokeExternalInterface_l, 
{firewallPolicy_0O -> permit_0O, firewallPolicy_5 -> spokeExternalInterface_0, 
firewallPolicy_l -> permit_O, firewallPolicy_6 -> hubExternalInterface_l, 
firewallPolicy_2 -> permit_O, firewallPolicy_7 -> hubExternalInterface_0, 
firewallPolicy_3 -> permit_0O, firewallPolicy_8 -> externalInterface_0} 


Display 13: Model for FullVPNSpec « BlockedIPSec. 
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sig router {} 
sig interface {chassis: router} 
pred EmptyCond () {} 


When Alloy tries to find a model for EmptyCond 
in a scope consisting of 50 routers and 50 interfaces it 
crashes! This is because the cross product of the set of 
all routers and chassis’ has 50x50=2500 pairs. Each 
subset of this product is a value of the chassis relation. 
Since there are 27°” subsets, there are that many pos- 
sible values to enumerate. We can now try splitting the 
scope and redefining the specification; see Diplay 16. 
Alloy returns a model of EmptyCond for the scope 
consisting of 25 hubRouters, 25 spokeRouters, 25 
hubRouterInterfaces and 25 spokeRouterinterfaces in sec- 
onds! Note that the scope still contains 50 routers and 
50 interfaces. But there are now “only” 2° x 265 = 
2!25° possible values of chassis relation, or a factor of 
2!750 less. The scope splitting heuristic has been 


pred internalSubnetAdvertisedToWan () 
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followed to structure the space of different routers and 
interfaces in the fault-tolerant VPN. 


Minimizing Number Of Quantifiers In Formulas 


Requirements containing quantifiers are trans- 
formed into Boolean form by instantiating quantified 
variables in all possible ways. The number of instantia- 
tions is the product of the number of instantiations of 
each quantified variable. The number of instantiations 
of each quantified variable is the size of the scope of 
that variable. In order to prevent the Boolean form from 
becoming excessively large, one can keep the number 
of quantified variables in a requirement as small as pos- 
sible. For example, consider the definition of Fire- 
wallPolicyRequirements show in Display 17 in which 
only two explicit quantifiers appear per requirement. 
Display 18 shows a more compact definition in which 
five quantifiers appear in a single requirement. 


{some r:legacyRouter, x:physicalInterface, y:physicalInterface, 
s:internalSubnet, e:externalSubnet | 


-chassis=r && 
.chassis=r && 
-network=s && 
-network=e && 
.routing=y. routing} 


aS KS 


Display 14: Advertising internal subnet if several conditions met. 
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Figure 6: Advertisement of internal subnet Y into WAN by legacy router. 


routing : = 


{physicalInterface_0 -> ospfDomain_O, 
physicalInterface_l -> ospfDomain_0O, 


AJ 
chassis : = 


{physicalInterface_0 -> legacyRouter_O, 
physicalInterface_1 -> legacyRouter_0O, 


a 
network : = 
{ 


physicalInterface_0 -> externalSubnet_3, 
physicalInterface_1 -> internalSubnet_l, 
1 


if 


Display 15: Code for expanded specifications (FullVPNSpec « internalSubnetAdvertisedToWan). 
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In both cases, the number of IPSec tunnels in the 
scope is 4 and the number of firewall policies 9. How- 
ever, there is a large difference in the size of the 
Boolean formula produced (for the entire VPN specifi- 
cation). In the first case, the formula contains 216,026 
clauses and 73,4262 literals, and the entire process 
(compilation to solution) took 2 minutes and 59 sec- 
onds. In the second case, the formula contains 601,721 
clauses and 2,035,140 literals and the entire process 
took 8 minutes and 19 seconds. 


Relationship To Previous Work 


IETF’s policy-based networking group [7] has 
similar objectives to ours. Its main contributions are 


sig hubRouter {} 
sig spokeRouter {} 


sig hubRouterInterface {chassis:hubRouter} 
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vendor-neutral information models and if-condition- 
then-action rules called policies. Information models 
define types of objects, their attributes and possible 
values. These models, while important from a software 
development standpoint, are orthogonal to solving fun- 
damental configuration management problems identi- 
fied in this paper. These problems would remain even 
if we were to use all components from a single vendor. 


Policy-based networking also does not enable 
any declarative representation of system logic such as 
Requirements 1-16. The if-condition-then-action rules 
are only procedural encodings of this logic. In effect, 
these rules have to do all the work of the Requirement 
Solver. This is a formidable undertaking. Furthermore, 


sig spokeRouterInterface {chassis:spokeRouter} 


Display 16: Splitting the scope and redefining the specification. 


pred FirewallPolicyRequirements () 


{(all t:ipsecTunnel | some pl:firewallPolicy | 


pl.protectedInterface = t.local && 


pl.prot = ike && 
pl.action = permit) && 


(all t:ipsecTunnel | some pl:firewallPolicy | 


pl.protectedInterface =t.remote & 


pl.prot = ike && 
pl.action = permit) && 


(all t:ipsecTunnel | some pl:firewallPolicy | 


pl.protectedInterface = t.local && 


pl.prot = esp && 
pl.action = permit) && 


(all t:ipsecTunnel | some pl:firewallPolicy | 


pl.protectedInterface = t.remote && 


pl.prot = esp && 
pl.action = permit) 
(no disj pl,p2:firewallPolicy | 


pl.protectedInterface=p2.protectedInterface && 
pl.prot=p2.prot && !pl.action=p2.action) } 


Display 17: Two explicit quantifiers per firewall requirement. 


pred FirewallPolicyRequirements () 


{(all t:ipsecTunnel | some pl.p2.p3,p4:firewallPolicy | 


pl.protectedInterface = t.local && 


pl.prot = ike && 
pl.action = permit && 


p2.protectedInterface = t.remote && 


p2.prot = ike && 


p2.action = permit && 


p3.protectedInterface 
p3.prot = esp && 
p3.action = permit && 


p4.protectedInterface = t.remote && 


p4.prot = esp && 
p4.action = permit) && 
(no disj pl,p2:firewallPolicy | 


t.local && 


pl.protectedInterface=p2.protectedInterface && 
pl.prot=p2.prot && !pl.action=p2.action) } 


Display 18: More compact version of Display 17. 
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verification with such procedural rules is impractical, 
and operations like configuration error diagnosis or 
fixing are not addressed. In our approach, the Require- 
ment Solver remains unchanged. It is only the require- 
ments or the scope that change. The Requirement 
Solver automatically adjusts to these changes and 
finds new configurations. Verification is another appli- 
cation of the Requirement Solver. 


For the same reason, it is incorrect to assume that 
just the use of high-level languages like Perl and 
Python, often used in network management, can solve 
above fundamental configuration management prob- 
lems. The hard part of reasoning from full first-order 
logic requirements still has to be programmed in these 
languages. It is this part that our approach automates. 


Previous papers [8, 9] formalized a restricted 
version of the second section’s requirements, in Pro- 
log. Prolog is based in definite clauses, hence it is not 
possible to use it to reason with full first-order logic 
constraints. Examples of these are “for every GRE 
tunnel there is an IPSec tunnel between associated 
physical interfaces that secures all GRE traffic” and 
“no two distinct interfaces on a router are on the same 
subnet.” The application of Prolog to system adminis- 
tration is thoroughly explored by Couch and Gilfix 
[10]. Related, widely used systems are Burgess’ 
CFEngine [11] and Anderson’s LCFG [12], but both 
have less expressive power than Prolog. These sys- 
tems also perform robust application of configuration 
to components, a problem outside the scope of this 
paper. Recently, the need for specifying and reasoning 
with constraints on configurations has been amply 
expressed [13]. These constraints require the expres- 
sive power of full first-order logic, therefore our 
approach can address this need. 


Summary, Conclusions & Future Directions 


This paper introduces the notion of a Require- 
ment Solver and shows how fundamental configura- 
tion management tasks can be naturally formalized 
using it. These tasks are configuration synthesis, 
requirement strengthening, component addition, con- 
figuration error diagnosis and configuration error fix- 
ing. The Solver has been inspired by, and is imple- 
mented in, the new logical system called Alloy. Alloy 
is based on the concept of model finding for full first- 
order logic in finite domains. Because of Alloy’s use 
of highly efficient satisfiability solvers, there is 
renewed optimism for efficient reasoning in this logic, 
especially for configuration management. Traditional 
first-order logic theorem provers address the harder 
problem of reasoning in infinite domains. The Solver 
is illustrated in the context of a realistic fault-tolerant 
VPN with remote access, by working out four of 
above tasks. Approaches for writing efficient specifi- 
cations are outlined. 


Alloy’s strength is efficiently sorting through 
complex, first-order logic constraints, provided scopes 
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are small. On a modern PC, it requires several hours to 
find complete configurations for all components in an 
8-site VPN: 8 spoke routers, 2 hub routers, 1 access 
server, | WAN router and associated interfaces, sub- 
nets, firewall policies, routing domains, GRE and 
IPSec tunnels. For context, the time taken by a human 
to reconcile Requirements 1-16 for all sites should be 
considered. Based on experiments of this paper, it is 
possible that Alloy be used from a traditional pro- 
gramming language to solve configuration manage- 
ment problems for networks of realistic scale and 
complexity. The heuristics of the “Efficient Require- 
ments” section could be programmed in the program- 
ming language. Other approaches for scalability are 
tuning satisfiability solvers to the networking domain, 
improving Alloy compilers, and using divide-and-con- 
quer approaches. 


One open problem is selecting the least cost 
change from the current configuration as required for 
Configuration Error Fixing. A configuration manage- 
ment problem, not discussed in this paper, is migration 
planning: in what order to configure components so 
that mission-critical invariants are never violated. For 
example, suppose the routing protocol on all routers 
has to be changed from RIP to OSPF. If the only 
method of accessing routers to perform this change is 
inband, then reconfiguring the first router to which the 
management station is attached will effectively isolate 
it from all others. This is because routing protocols 
compute routes to routers, but since OSPF and RIP 
processes do not exchange information with each 
other, the first router will not be able to compute 
routes to others. The problem of the order in which to 
reconfigure components is fundamentally the problem 
of planning in artificial intelligence. The application 
of satisfiability solvers to this problem has been 
shown by Selman and Kautz [14]. 
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Visualizing NetFlows for Security at 
Line Speed: The SIFT Tool Suite 
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ABSTRACT 


The first step in improving Internet security is measurement — security events must be made 
visible. The irony in making this happen is that there is no lack of security measurement data, in 
fact, quite the opposite. However, making security manifest faces a major challenge: the large 
volume and multi-dimensional nature of security data typically obscures valuable security events. 
NCSA has developed a suite of tools that solves this problem and is making this software available 
to the Internet community. 


We present two visualization tools,’ (1) NVisionIP and (2) VisFlowConnect-IP. Both of these 
tools have been developed based on system administrator requirements, their design peer-reviewed in 
security research forums, and usability testing is in process. These tools both present large volume 
complex data transparently to system administrators in simple intuitive visual interfaces that support 
human cognitive processes. NVisionIP visually represents the state of all IP addresses on large 
networks on a single screen window (we use a Class B address space as the default) with capabilities to 
filter and drill-down to subnets and individual machines for details-on-demand. VisFlowConnect-IP 
visually represents flows between intemal network IP hosts and the Internet showing who is connecting 
with whom with capabilities to filter and drill-down to subnets and individual machines for details-on- 
demand. NVisionIP and VisFlowConnect-IP can be used individually or in unison for correlating 
events. This work is distinguished from others in that these are the first Internet security visualization 


tools to be freely available on the Internet and deployed in large production environments. 


Introduction 


Organizations use computer network infrastruc- 
tures that hold a vast amount of information for sys- 
tem administrators and security engineers. There are 
typical logs common to most computer networks, but 
the systems are often large and dynamic, making it 
difficult to extract knowledge from the sea of informa- 
tion. Individually, each system log can be massive, 
causing operator overload. When overload occurs, 
security events can slide by unnoticed. Overload can 
also cause operators to disregard alarms due to high 
false positive rates. Even in homogeneous infrastruc- 
tures, solutions from a single vendor fail to scale to 
medium or large networks. However, the problem is 
compounded because most organizations have net- 
work infrastructures from multiple vendors. 


We have developed Security Incident Fusion 
Tools (SIFT) [9], an integrated suite of tools for evalu- 
ating the security of an entire computer network on a 
single screen. We address the need to discover security 
incidents that currently go undetected by security 
operations systems. Specifically two SIFT tools, (1) 
NVisionIP and (2) VisFlowConnect-IP, leverage 
human visual cognitive abilities to process log data 
into knowledge for situational awareness of network 
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tion, and Commercialization Center (TRECC) and the Nation- 
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security. It is estimated that human beings can visually 
process a screen of information at 150 Mbits per sec- 
ond [10], with the ability to discriminate relatively 
minor shifts in color, shape, and motion. By present- 
ing network data visually, it can be scanned quickly, 
patterns in complex data rise to the surface, and infer- 
ences become intuitive. Once a security professional 
becomes familiar with the normal appearance of the 
network being monitored, it is much easier to spot 
attacks including new so-called “zero-day attacks.” 
The tools are designed to give security engineers situ- 
ational awareness of an entire network in order to help 
them determine when a network is under attack, what 
is being attacked, and what form the attack is taking. 


The remainder of the paper is organized as fol- 
lows: The next section discusses NetFlows data man- 
agement and introduces the first tool in the SIFT suite: 
CANINE. Subsequently we present the two SIFT visu- 
alization tools — NVisionIP and VisFlowConnect- 
[Pand close with a summary and on-going future work. 


Data Management 


NetFlows Source Data 


While this paper focuses on visualization, we 
would be remiss if we did not address data manage- 
ment since it is arguably the greatest obstacle in realiz- 
ing any scalable visualization system. We address the 
challenge of processing high-bandwidth data streams 
by instrumenting networks with distributed NetFlows 
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sensors and then combining this sensor data into a uni- 
fied format. While in the recent past NetFlows were 
solely router-based, PC-based NetFlow — sensors 
(Argus) make this a feasible solution for most organi- 
zations. The first tool in our suite is a NetFlows con- 
verter/anonymizer called CANINE which can handle 
different NetFlows formats so independent implemen- 
tations can be interoperable with SIFT visualization 
tools. NetFlow logs have proven to be the appropriate 
granularity to process heavily loaded networks and 
high bandwidth connections (Gb/s) in near-real-time 
(five minute monitoring windows). 


A network flow is defined as a sequence of pack- 
ets that are transferred between two endpoints within a 
certain time interval. The endpoints are identified at 
the network layer by IP addresses and at the transport 
layer by port numbers. In addition to data format dif- 
ferences, there are other interoperability problems in 
practical NetFlows implementations: 

® Cisco NetFlows are defined as unidirectional 
and generated through intelligent flow cache 
management, which contains a set of special- 
ized algorithms [4]. 

e Argus NetFlows are defined as bidirectional 
containing two distinct sub-flows, one in each 
direction [2]. 

® Cisco and Argus NetFlow formats have differ- 
ent fields (e.g., flags etc.) [3, 5]. 


For a more detailed comparison between differ- 
ent NetFlows formats see [15]. 


CANINE 


With the increased use of NetFlows for security 
monitoring and the fact that NetFlows come in differ- 
ent and incompatible formats, we have developed 
CANINE (Converter and ANonymizer for Investigat- 
ing Netflow Events) [7, 8] which can be downloaded 
from http://security.ncsa.uiuc.edu/distribution/Canine- 
DownLoad.html. CANINE allows tools designed for 
a specific type of NetFlows to be interoperable with 
any NetFlow format. CANINE consists of the two 
main modules: (1) the CANINE GUI and (2) the con- 
version/anonymization engines. For the purposes of 
this paper we will only discuss the conversion engine 
(for information about the anonymization engine see 
[7, 8]). The CANINE GUI accepts user input to iden- 
tify the NetFlow file for conversion, sends the request 
to the processing engine which performs the conver- 
sion to the newly specified output file, and lastly sum- 
marizes the results of the performed actions in a pop- 
up window. At present CANINE supports conversion 
to/from Cisco version 5/7, Argus, NFdump, and our 
own NCSA internal NetFlows format. Future formats 
to be included in CANINE include Cisco version 9 
and the future IETF IPFIX standard. 


Network Instrumentation 


With the development of high-speed network in- 
frastructure has also come the need for high-speed 
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security — security at line speed — for current 2005 net- 
works this is 4 GB/s at the edge and higher within the 
core [15]. Unfortunately, high network bandwidths 
present special problems for security monitoring. 


The first challenge is the streaming nature of 
security sensors. It is important to note that security 
sensors generate streaming data and not batch log 
files. Since streaming analysis is an open research 
question, security systems typically create batch log 
files by collecting streaming data over defined time 
periods. However, depending on the network size and 
traffic volume these log files can become large and 
difficult to handle. Tuning is required to determine the 
best time period of analysis to match the preferred log 
size to the network size and traffic volume. Creating 
logs over longer time intervals may risk losing Net- 
Flows records upon high transmission rates from over- 
flow or blocking. 


The second challenge is observation point. Secu- 
rity cannot be measured where it is not observed thus 
sensors need to be placed to cover the entire network 
space. Typical deployment for NetFlows includes the 
border router for Internet traffic and Argus sensors for 
internal network observation. There are blind spots 
from VLANs and switched networks which do not 
leave IP (network layer) traces — future sensors based 
on S-Flows are developing to address this gap. 


The third challenge is CPU speed to generate and 
process NetFlows at line speed. As routers have 
increased speed, monitoring techniques have shifted to 
sampling NetFlows. While sampling is statistically 
sufficient for network planning, it is not a good idea 
for security analysis. NetFlow records are created by 
sampling packets (not flows), letting the majority of 
the packets go unnoticed, which may lead to missing 
important security events. A possible justification for 
sampling is that an attack may be high traffic volume, 
at least part of which may be captured with high prob- 
ability (such as high-volume denial-of-service attack 
or indiscriminate scanning by propagating worms and 
viruses). A preferred approach we recommend for 
security at line speed is the parallel processing Net- 
Flows in a distributed manner. Instead of instrument- 
ing only the high-speed border router that may only be 
able generate sampled NetFlows, instead instrument 
all the routers feeding into the border router. This 
technique effectively relieves the load on each flow 
collector so that it will not be over subscribed. The 
drawback is that multiple flow collectors are required 
and NetFlows records from different routers must be 
merged to eliminate duplicate flows (the same flow 
that passes through multiple routers). 


NetFlows Visualization Tools 


Design By Requirements 


We firmly believe that the first step to improve 
Internet security is by measurement. Measurement 
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allows one to accurately assess the degree of the prob- 
lem at a specified time and then further measurements 
track whether solutions are having the desired effect. 
However, not all measurements are equal, users have a 
mental model based on experience and tools should be 
designed to enhance and augment these mental models 
for the most effective results [16, 17, 18]. 


For this work, we did two important things often 
neglected from security tool design: (1) taking time to 
work with security engineers in their operational pro- 
duction environment in order to learn their mental 
models and thus tool requirements and (2) the capabil- 
ity to design new visualization models from scratch to 
meet these requirements without having to incorporate 
legacy constructs. The results have been very satisfy- 
ing in that most security engineers who view our visu- 
alization tools for the first time immediately begin 
inferring hypotheses based on the content displayed. 


To briefly summarize the major findings from 
our requirements analysis there are two primary find- 
ings. First, security engineers need to answer ques- 
tions such as these posed by upper management: What 
is the state of the network? Is the network being 
attacked? How is the network being attacked? Who is 
attacking the network? While these may appear to be 
basic questions, the answers are not immediately 
available using current security tools and when avail- 
able after much analysis the answers are complex. 
Visualization provides a rich representation to help 
answer these questions concisely. 


Second, security engineers have mental models 
based on their experience with the network infrastruc- 
ture, knowledge of people within the organization, and 
security expertise learned over many years. While 
most tenets of information visualization design are use- 
ful in designing within our specific security domain, 
we did find that leveraging the mental model of secu- 
rity engineers caused us to break some of these consen- 
sus rules (after much consternation). Instances when 
the security engineer mental model overrides informa- 
tion visualization design best practices are highlighted 
in our discussion of each of our visualization tools. 


NVisionIP 


Our first and most mature security visualization 
tool is NVisionIP [1, 6] which we designed to answer 
the question: What is the state of the network? 


Figure | shows the Galaxy view of NVisionIP 
which can be downloaded from http://security.ncsa. 
uiuc.edu/distribution/N VisionIPDownLoad.html. The 
Galaxy view represents an entire Class B IP address 
space (in this single window!) as a matrix with subnets 
along the horizontal axis and hosts along the vertical 
axis. Each IP address is represented as a dot (actually 
four pixels) and the state of each IP address is repre- 
sented with color or shape as determined by the user in 
the color and shape legend. Two magnification options 
are available to see the IP addresses: linear and fisheye. 


Visualizing NetFlows for Security at Line Speed: The SIFT Tool Suite 


NVisionIP allows the security engineer is to load 
one (or multiple) NetFlow files and perform visual 
queries. NVisionIP has taken all the possible NetFlow 
database query combinations and hard coded them into 
the tool as drop-down and point-and-click commands. 
A user would typically start with primary queries such 
as how bytes per IP address or how many connections 
per IP address. A filter then allows the user to select 
secondary queries to view only source or destination 
traffic, different protocol (IP, UDP), and different ports 
(destination or source ports, specific ports or collec- 
tions of ports) or any combination thereof. 


At the Galaxy view, NVisionIP can identify large 
or small levels of traffic as measured in bytes (based 
on expectations for the class of machine — laptop or 
server). This may indicate malware is being served 
to/from a machine or the machine is involved in a 
denial-of-service event. Worm and virus scans as indi- 
cated by number of connections can also be easily 
detected based on variance from expected levels. 


There are aspects of the Galaxy view design that 
are contrary to information visualization best prac- 
tices: the IP address space is laid out logically in 
matrix space without organizing IP addresses into 
known classes or enlarging the part of the IP address 
space with more activity (thus patches of white space 
or inactive IP address space appears). This design was 
intentional to retain security engineer knowledge of 
the IP address space based on logical numbering for 
subnets/hosts and mental mapping between logical 
addresses (e.g., cluster compute nodes with contiguous 
IP addresses) and physical locations (IP subnets are 
usually physically located in the same area such as a 
building floor etc.). The white space of inactive IP 
addresses actually has other advantages and is not 
wasted space — any traffic activity shown there is 
anomalous (unallocated address space that should 
have no legitimate traffic). 


While an overall view is important, it is of lim- 
ited use without the ability to drill down to find more 
detailed information when something interesting is 
identified. Figure 2 shows the drill-down levels of 
NVisionIP which are activated with a mouse click and 
a drag over a region of interest. These levels are the 
(1) Small Multiple View and (2) Machine View. 


The Small Multiple View allows the user to 
quickly scan and compare traffic activity across sub- 
nets on many machines simultaneously. Each machine 
is a box with two sets of histograms, an upper set of 
histograms representing traffic on well-known ports 
and a lower set of histograms representing traffic on 
ports over 1024. The well-known ports are color- 
coded in a user legend. The ports over 1024 are 
ordered from most active to least active (top N ports). 
Note that no numbers are shown in the small multiple 
view, this view is designed for the user to identify 
activity of interest and then drill-down for raw data 
details on-demand. 
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Figure 1: NVisionIP Galaxy view of an entire Class B IP address space. 
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Figure 2: The three levels of NVisionIP (top to bottom): (1) Machine view, (2) Small multiple view, and (3) Galaxy view. 
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Figure 3: VisFlowConnect-IP: Main view. 





, es ~ = 
te femme seer 

Meme meet datenet mete Means Tate nee a : 

wee nd.’ 


tee ® wee Ney 


Lone nee mee oe Dea ements 





eines Osa bitte 
Cemain View 
L os ee OORT eee a 


internal View 
Figure 4: Drill-down layers of VisFlowConnect-IP: left Domain View and Bright Internal View. 
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At the Small Multiple View, NVisionIP has been 
used to quickly identify anomalous services that vio- 
late security policy such as unauthorized services (web 
or mail server) and exposed services that need to be 
patched or otherwise protected due to vulnerabilities. 


If a user sees a machine with interesting traffic activ- 
ity in the Small Multiple View, they may click on that 
block to drill-down to the Machine View. The Machine 
View organizes all the data from a particular machine in 
multiple tabs — each tab showing aggregate activity in an 
upper histogram and source/destination directional flows 
in two lower histograms. Note in the Machine View each 
histogram is fully labeled with port numbers and traffic 
level. At the lowest level, the raw NetFlows data for that 
machine is available for inspection in this Machine View 
as a tab. At this lowest level Machine View, details of 
most security events can be revealed. 


The overall effect of using NVisionIP (with its 
interactive drill-down levels) is that relationships 
between aggregate network activity and individual 
machine activity can be more easily discovered and 
comprehended by human operators — providing situa- 
tional awareness of network system state. Future work 
is progressing on optimizing Galaxy View animation 
to show IP address state changes over time as well as a 
difference view to visually compare current network 
traffic versus benchmark network traffic. 


VisFlowConnect-IP 


The second security visualization tool in the 
SIFT tool suite focuses on answering the question: 
Who is connecting to whom on the network? This 
basic question has been attempted in the past with 
topology-based diagrams based on network infrastruc- 
ture, however, these results were either (1) not provid- 
ing information relevant to real-time traffic or (2) not 
scalable since representing network traffic over time 
produces obscured lines in relatively short order. We 
solve both of these problems with VisFlowConnect-IP 
{12, 13, 14] which is available at http://security.ncsa. 
uiuc.edu/distribution/VisFlowConnectDownLoad.html . 


VisFlowConnect-IP is a security visualization 
tool based on the parallel axes concept drawn from 
data mining. It is a complementary tool to NVisionIP 
since it visualizes the same NetFlows source data — the 
design similarities will become apparent in the follow- 
ing description. VisFlowConnect-IP allows a user to 
visually assess the connectivity of large and complex 
networks (in a single window!) by providing a main 
view of the network with filter and drill-down views 
that provide more details on-demand. The three views 
of VisFlowConnect-IP are: (1) Main, (2) Domain, and 
(3) Internal. The Main View is shown in Figure 3 with 
the Domain and Internal views shown in Figure 4. 


The VisFlowConnect-IP Main View utilizes the 
parallel axis view with the left-most and right-most 
vertical axes representing the external domains and the 
center vertical axis representing host IP addresses 
within the internal edge network domain (See Figure 
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3). Lines connecting external domains and internal 
hosts represent directional data flows, with line dark- 
ness being proportional to the logarithm of the volume 
of data transferred. VisFlowConnect-IP can _fil- 
ter/highlight flows to certain hosts or traffic on spe- 
cific ports and protocols using a filter drop-down 
menu and selection boxes on the main view. Ports 
indicated in the selection boxes are represented in dif- 
ferent colors within the network traffic or may be iso- 
lated from network traffic for focused analysis. The 
overall effect is visualization of traffic into-an-edge- 
network-from-the-Internet and traffic out-from-an- 
edge-network-to-the-Internet. 


Figure 4 shows the two drill-down views within 
VisFlowConnect-IP. While we would have liked to 
represent each individual external host IP address con- 
necting into the internal edge network symmetrically 
on both the left-most and right-most axes, this is not 
possible due to scalability. Preliminary measurements 
of NCSA’s network showed over 100,000 different IP 
addresses commonly appeared in the NetFlow files we 
wished to visualize and this is too many for the verti- 
cal line pixel space of a single window without 
scrolling. Instead we implemented a drill-down 
Domain View which is invoked by the user clicking on 
a drop-down menu while having an external domain 
highlighted on the vertical external domain axis. The 
resulting Domain View is a mirror image of the Main 
View except it only shows traffic within the high- 
lighted external network domain to/from the internal 
edge network. This has turned out to be very valuable 
since typically hackers “own” entire subnets or even 
“own” entire network domains so it is common to see 
malicious activity captured within a Domain View. 


Figure 4 also shows the drill-down Internal View 
which is invoked as a toggle button on the Main View. 
While monitoring for external Internet hacker activity 
is sexy, we have found this Internal View very useful 
since it shows only traffic that both sources and sinks 
within the internal edge network. There are only two 
vertical lines in this view, internal edge network IP 
addresses are ordered symmetrically in a mirror image 
on the left-most and right-most axes (no middle axis). 
This Internal View has helped security engineers 
determine important security events like the initial 
source of a worm infection which infiltrated the edge 
network from the inside, and the insider attacks from 
those misusing privileged access. 

The VisFlowConnect-IP Main View has a time 
axis at the bottom which is used to solve the scalabil- 
ity problem we referred to as the major challenge for 
this tool. The user loads a NetFlow file for visualiza- 
tion and then may select multiple filters to determine 
how this traffic is to be represented in animation 
including intensity, byte size, and a sliding time win- 
dow. The sliding time window provides scalability by 
only representing traffic within the window and ignor- 
ing traffic outside the window. Thus the sliding time 
window can be adjusted to any size network and any 
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traffic volume — the general rule for clear viewing is 
the more traffic the smaller the sliding time window. 
The window size itself is represented to the user by a 
red box (where the length of the red box is propor- 
tional to window size) that travels along the time axis 
as the traffic is animated (as shown in the Domain and 
Internal Views within Figure 4). 


VisFlowConnect-IP has also implemented a filter 
language using real expressions that is beyond the 
scope of this paper [12]. With this filter language 
capability, VisFlowConnect-IP can create mechanisms 
for storing/retrieving filter profiles. These profiles can 
store customized filters that remove “uninteresting” 
information from view-thus leaving only the more 
security relevant data to be displayed. 


Summary 


Visualization is the future of security monitoring 
and NetFlows are the source data for high-speed net- 
works. In this paper we marry security visualization 
with NetFlows by presenting the SIFT suite of tools 
along with accompanying techniques for security at 
line speed. The goal is to enable security engineers to 
go beyond binary/text command line log file analysis 
toward real-time network security situational aware- 
ness. A growing community of researchers has formed 
on security visualization, see [11] for more information. 


The three specific tools of the SIFT suite pre- 
sented in this paper (CANINE, NVisionIP, and Vis- 
FlowConnect-IP) are available for download at the 
URLs provided in the text. We are currently conducting 
usability tests with human subjects to quantify the util- 
ity of these tools and preliminary results from these tests 
are very promising. We intend to go open source with 
these tools after the software is stable, at present we are 
still developing the software with new versions posted 
on the corresponding webpages. We enthusiastically 
invite feedback from users about the use of these tools. 
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Interactive Traffic Analysis and 
Visualization with Wisconsin Netpy 


Cristian Estan and Garret Magin — University of Wisconsin-Madison 


ABSTRACT 


Monitoring traffic on important links allows network administrators to get insights into how 
their networks are used or misused. Traffic analysis based on NetFlow records or packet header 
traces can reveal floods, aggressive worms, large (unauthorized) servers, spam relays, and many 
other phenomena of interest. Existing tools can plot time series of pre-defined traffic aggregates, 
or perform (hierarchical) “heavy hitter” analysis of the traffic. 


Wisconsin Netpy is a software package that goes beyond the capabilities of other existing 
tools through its support for interactive analysis and novel powerful visualization of the traffic 
data. Adaptive sampling of flow records ensures that the performance is good enough for 
interactive use, while the results of the analyses stay close to the results based on exact data. 
Among the salient features of the package are: hierarchical analyses of source addresses, 
destination addresses, or applications within aggregates identified by user-defined filters; time 
series plots that separate the traffic into categories specified with ACL-like syntax at run time; 
interactive drill-down into analyses of components of the traffic mix; “heatmap” visualization of 
traffic that describes how two “dimensions” of the traffic relate to each other (e.g., which sources 


send to which destinations, or which sources use which service, etc.). 


Introduction 


The unrestricted packet communication sup- 
ported by the Internet offers immense flexibility to the 
endhosts in how they use the network. This flexibility 
has enabled the deployment of new applications such 
as the web long after the IP protocol has been stan- 
dardized and has contributed significantly to the suc- 
cess of the Internet. On the other hand, network opera- 
tors want to monitor and to some extent control how 
their networks are used. Firewalls, network address 
translation, and traffic shaping boxes offer a degree of 
control that helps keep networks manageable. But 
even within the constraints of the policies imple- 
mented through these devices, the network traffic is 
very variable and traffic monitoring is necessary. 


An analysis of network traffic can reveal impor- 
tant usage trends such as the application mix and the 
identity of the heaviest traffic sources or destinations. 
Sometimes these analyses can reveal misuses of the 
network: compromised desktop computers turned into 
spam relays, remote computers scanning the network 
for vulnerabilities, network floods directed against a 
single victim, or caused by a worm trying to spread 
aggressively. It is often the case that the analysis is 
urgent because it is carried out to explain a degrada- 
tion in network service. It is also often the case that 
the network administrator does not know in advance 
which ports or IP addresses to focus on and he goes 
through an iterative process before being able to find 
convincing evidence for the cause of the problem. For- 
tunately there are many traffic analysis and visualiza- 
tion tools to assist the network administrator in the 


task of exploring and understanding the traffic carried 
by their network. Wisconsin Netpy is a new and pow- 
erful addition to this large family. 


Related Work 


Tobias Oetiker’s MRTG [12] is an early traffic 
visualization tool widely used by network administra- 
tors to track the volume of IP traffic based on SNMP 
counters provided by routers and switches. His RRD- 
tool [12] provides support for visualizing time series 
plots of arbitrary data and it is actually used by all 
traffic visualization applications discussed in this sec- 
tion. Jeff Allen’s Cricket [1] takes MRTG’s idea one 
step further by allowing the user to track a large num- 
ber of variables (say the traffic of various links in the 
network) using a scalable config tree. These tools offer 
visual information only about the volume of the traf- 
fic, but nothing on the composition of the traffic mix. 


The NetFlow flow records generated by routers 
are an information source much richer than the SNMP 
counters. Toolkits such as OSU flow-tools [7] and 
SiLK [8] have tools for manipulating and analyzing 
NetFlow data. Typical analyses allow finding the top 
sources and destinations of traffic. cflowd [2] is a 
related package that also supports various types of 
traffic matrices. Dave Plonka’s FlowScan [13] and 
Cristian Estan’s AutoFocus [5] also provide time series 
of various predefined categories of traffic represented 
within the traffic mix. The supercomputing community 
is also interested in IP traffic visualization, and its 
members have written tools such as”’the spinning cube 
of potential doom” [11] and NVisionIP [10]. 


AutoFocus and Ryo Kaizaki’s aguri [9] employ a 
novel type of analysis related to top k reports called 


19th Large Installation System Administration Conference (LISA °05) 177 


Interactive Traffic Analysis and Visualization with Wisconsin Netpy 


hierarchical heavy hitters or traffic cluster analysis [6, 
3]. This type of analysis is used extensively by Netpy. 


Network administrators are often interested in 
the largest sources or destinations as measured in 
bytes, packets or flows. One important observation is 
that while existing tools give the exact traffic for these 
heavy hitters, the user can often accept small errors. In 
fact such small errors are already present if one uses 
sampled NetFlow. Netpy exploits this observation by 
sampling flow records to speed up traffic analysis and 
to reduce disk usage. For measuring the traffic in bytes 
or packets we use the “smart sampling” of flow 
records introduced by Duffield, et al., [4]. 


Traffic Analysis With Netpy 


The user can direct Netpy to perform traffic 
analyses through a graphical user interface or through 
a console that supports interactive queries as well as 
scripts. All analyses use an intermediary database of 
flow records (see “The Structure of netpy”’ for more 
details about Netpy’s structure). But what kind of traf- 
fic analyses can one perform with Netpy? That’s the 
question we answer through the rest of this section. 


Time Series Plots With User Defined Categories 


Time series plots are an easy to read visual repre- 
sentation of the traffic. Existing tools such as FlowS- 
can and AutoFocus allow the network administrator to 
define various traffic categories based on port num- 
bers or network prefixes and have them plotted with 
separate colors. This way the plots reveal information 
about the cause of various spikes. Furthermore with 
separate plots measuring the traffic in bytes, packets, 
(and flows in a future version) the user will be able to 
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detect not only large floods, but also scans that gener- 
ate many flows, but not many bytes. 


Netpy also supports these types of time series 
plots. The user can specify the categories using an 
ACL-like syntax: each rule specifies a source and des- 
tination prefix, protocol number and source and desti- 
nation port range; flows are mapped to the category 
associated with the first rule they match. For FlowS- 
can and AutoFocus the user needs to specify the cate- 
gories of interest before the NetFlow data is 
““imported,”” whereas with Netpy the user specifies the 
categories at run time and it is quicker to recompute 
the plot after the user changes the ACL rules defining 
the categories because the analysis relies on the data- 
base, not on the large NetFlow files. 


The Scope of Traffic Analysis 


For time series plots, and all the other analyses, 
one needs to define which NetFlow records constitute 
the input to the analysis. Existing toolkits often allow 
the network administrator to configure separate traffic 
reports for separate links. Netpy separates NetFlow 
data into different links as data is imported into the 
database. When running an analysis, the user specifies 
which links’ traffic to work with. The user also speci- 
fies the time interval of interest to the analysis. 


The user can also specify a filter to apply to the 
data matching the previous two criteria. The filter con- 
sists of one or more rules similar to router ACLs (each 
rule specifies a source address prefix, destination 
address prefix, source port range, destination port 
range, and protocol number) and flow records that 
don’t match any of the rules in the filter are not con- 
sidered in the analysis. The GUI’s interactive drill- 
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Figure 1: Hierarchical heavy hitter analysis on the sources of the traffic. Indentation is used to highlight prefixes in- 
cluding each other. The analysis finds the appropriate granularity based on the current traffic: 208.0.0.0/4 is a 
sixteenth of the address space and 145.89.5.212/32 is a single IP (all addresses are anonymized), but they are 


both reported because their traffic is above the threshold. 
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down feature works by setting the filter to select only 
the traffic of interest. 


Hierarchical Heavy Hitters 


The network administrator cannot always know 
in advance what port numbers or IP prefixes will dom- 
inate the traffic, so forcing her to specify in advance 
the ACL rules defining the categories doesn’t always 
work, This is especially true after one drills down into 
a small, unfamiliar portion of the traffic mix. A tradi- 
tional solution to this problem is to use “top K 
reports”: one computes the traffic of each source 
address and reports the top K (say top 20). A related 
solution is the “heavy hitter report” which reports all 
sources whose traffic is above a given threshold in the 
data analyzed (say more than 1% of the total traffic). 
A problem with both these solutions is that they tell us 
nothing about sources that send little traffic: if for 
example we have a prefix with many small sources 
that nevertheless add up to a large portion of the traffic 
(a large modem pool), we would want to find out 
about their behavior. Netpy relies on the “hierarchical 
heavy hitter” algorithm that finds not just individual 
addresses, but also prefixes whose traffic is above a 
certain threshold specified as a percentage of the traf- 
fic being analyzed. This algorithm has the property 
that it not only identifies the prefixes generating sig- 
nificant traffic, but it also automatically finds the right 
prefix lengths to use when describing various portions 
of the traffic. 


The hierarchical heavy hitter algorithm works as 
follows: first it reports all individual IP addresses 
whose traffic is above the threshold, next it aggregates 
the remaining traffic at the /31 level and reports any 
prefixes that are above the threshold, next it aggre- 
gates the remaining traffic at the /30 level and so on 
until it reaches the root of the IP address hierarchy. 
The criterion for reporting more general prefixes is 
that the difference between their traffic and the traffic 
of more specific prefixes already reported is more than 
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the threshold. However, when Netpy reports such a 
prefix, it reports its total traffic not the difference 
between its traffic and that of more specific prefixes. 
Figure | shows the result of a hierarchical heavy hitter 
analysis on the source addresses in the traffic mix 
using a threshold of 5%. Through the threshold the 
user can control the level of detail: with a lower 
threshold, more prefixes are reported, with a higher 
threshold the user gets a coarser view. 


The same type of hierarchical heavy hitter 
approach applies to destination IP addresses too. The 
approach actually generalizes to any hierarchy we can 
define on one or more of the packet header fields 
present in the flow record. To capture information 
about the applications in use, Netpy defines the fol- 
lowing hierarchy: the first level below the root divides 
the flows by protocol, the second level divides the 
flows by source port into flows originating from low 
ports (0 to 1023) usually used by servers and high 
ports (1024 to 65535) usually used by clients, the third 
level divides the flows by actual source port value and 
the fourth level divides them by source and destination 
port value. The analysis will pick the granularity of 
the results based on the actual traffic. For example if 
there is a large TCP connection (e.g., a huge backup), 
the amount of traffic between its source port and desti- 
nation port will be reported. If there is a source port 
used by many small connections (e.g., web traffic on 
port 80), the total traffic coming from port 80 will be 
reported. If there is no dominant source port, but the 
source ports used are in the high port range (e.g., traf- 
fic coming from a network of typical desktop comput- 
ers). An example of this type of analysis is shown in 
Figure 2. 


Bidimensional Analysis 


The analyses looking at simple hierarchies such 
as the ones above can tell you that TCP port 80 and 
UDP port 53 generate a lot of traffic, and they can also 
tell you that servers A and B generate a lot of traffic, 
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Figure 2: Hierarchical heavy hitter analysis on the application hierarchy. Based on the traffic the analysis picked to re- 
port the traffic for the entire UDP protocol, for high and low TCP source ports, for individual TCP source ports 80 
(web) and 139 (netbios) and for two UDP source and destination port pairs (used by a database application). 
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but you won’t be able to tell which one is a web server 
and which one is a DNS server. With Netpy’s “unidi- 
mensional” reports the user can look at these hierar- 
chies in isolation. Netpy also has ‘bidimensional”’ 
reports that look at two hierarchies at once: Netpy 
computes the relevant categories for both dimension 
and reports a crossproduct of the results — for every 
pair of categories from the opposite hierarchies, the 
traffic matching both categories is reported. For the 
example above, if we run a bidimensional analysis on 
the application and source address dimensions, the 
application dimension will pick (protocol=TCP,source 
port=80) and (protocol=UDP,source port=53) as rele- 
vant categories and the source address will pick 
(source address=A) and (source address=B). The bidi- 
mensional report will have the traffic of the following 
four combined categories of traffic: (proto- 
col=TCP,source port=80,source address=A), (proto- 
col=TCP,source port=80,source address=B), (proto- 
col=UDP,source port=53,source address=A), and (pro- 
tocol=UDP,source port=53,source address=B). 


In the GUI, the two dimensions of a bidimen- 
sional report are the two sides of a square and the cate- 
gories defined within individual dimensions are repre- 
sented as small segments on the sides of the square. 
The rectangles within the square represent the com- 
bined categories. The darkness of the rectangles indi- 
cates the amount of traffic of the combined category 
with darker shades indicating more traffic. The GUI 
displays the actual amount of traffic in any combined 
category when the user moves the mouse over the cor- 
responding rectangle. Using darkness to convey the 
intensity of traffic (or other data) is known as the 
“‘heatmap”’ representation. Figure 3 shows a bidimen- 
sional analysis with the application hierarchy as the 
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horizontal dimension and the source IP address hierar- 
chy as the vertical dimension. 


The bidimensional reports do not capture all the 
information present in the (textual) multidimensional 
reports used by AutoFocus that consider all five 
packet header fields at the same time. The advantage 
of these bidimensional reports is that they can be com- 
puted much faster than the multidimensional reports 
and yet they can convey much of the information 
present in a multidimensional report. 


Structure of Netpy 


Netpy has four main parts: the database, the anal- 
ysis engine, the console and the GUI. The role of the 
database is to store preprocessed NetFlow records and 
deliver the records selected for the current analysis to 
the engine. The analysis engine runs the hierarchical 
heavy hitter algorithm, and all other analysis algo- 
rithms supported by Netpy. The console is a text based 
interface to the analysis engine and the only interface 
that allows the network administrator to update the 
database. The GUI is an interface that visualizes the 
traffic analysis results and helps the user navigate the 
traffic data. 


The Database 


The Netpy database is an intermediary represen- 
tation of the NetFlow flow records. The aim of the 
database is to preprocess the flow records in a way 
that ensures that when the user asks for an analysis, 
one has to read from disk the minimum amount of 
data needed to compute the result. Quick analyses are 
important for interactive exploration of the traffic mix. 
The entire database manipulation code is written in C 
and it links against the flow-tools library. 
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Figure 3: Bidimensional Hierarchical heavy hitter analyses on the application hierarchy and source IP address hierarchy. 
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Database Structure 


The Netpy database is actually a hierarchy of 
files with simplified flow records. This format has the 
following four main advantages over just storing Net- 
Flow records directly: data reduction and better con- 
trol over disk usage through adaptive sampling when 
there are too many flow records; storing flow records 
for different links in different files; using separate files 
for time bins that make it easier to only read in the 
records of flows active during the selected time inter- 
val; more compact flow records with fewer fields. 


Console 






Analysis 
Engine 


Figure 4: Netpy’s modules. 
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In the links.conf configuration file the user can define 
any number of links. The file has a list of rules with 
the NetFlow fields in this table. Each flow record is 
mapped to the link associated with the first rule it 
matches. For all fields, a **’ in the rule matches all 
possible values. 



























Netpy groups flow records into “links” based on 
the links.conf configuration file that uses the fields 
from Table 8 to select the link a flow record belongs 
to. This allows the network administrator to separate 
traffic carried on various links of a router that is never- 
theless reported together. The flow records corre- 
sponding to different links are then stored in separate 
directories. Thus when the user runs an analysis on 
only one of the links, we don’t have to go through all 
flow records, but only read those mapping to that link. 

The links.conf file also specifies a cap for each 
link on how much hard disk space an hour’s worth of 
traffic can take. When the number of flow records 
exceeds the allotted space, Netpy applies sampling 
using a rate that ensures that the disk usage stays 
within budget. The smaller the disk usage allowed, the 
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more aggressive the sampling has to be and the less 
accurate the results of the analyses will be. See the 
section on sampling algorithms for results on the 
amount of error introduced by sampling. 


Each directory corresponding to a link contains 
individual files with flow records, each representing a 
five minute time bin. For the current version of Netpy, 
this does limit the time intervals. The user can only 
request analyses on multiples of five minutes, but once 
the user specifies the interval, we can read in the right 
flow records by just reading the files representing the 
five minute bins included in the interval. Some of the 
original NetFlow flow records can span two or more 
bins. We handle these by splitting them and storing the 
resulting records in their respective bins. This splitting 
of records results in an increase of only 2% in the 
number of records stored in the database which is a 
price worth paying. 

The flow records in the database contain only the 
fields used in the analyses: source and destination IP 
address, protocol, source and destination port and byte 
and packet count. We need not store timestamps 
because the file a flow record is in identifies which 
five minute time bin it belongs to. We also discard 
most of he fields from Table 8 because the useful 
information they hold has already been incorporated in 
the choice of the link the flow record is mapped to. 
This way the size of a flow record is reduced from 60 
bytes to 21. 


Reading from the Database 


The analysis engine specifies what data to select 
for the analysis. This specification has four parts: the 
list of links to include, the time interval, a filter, and 
whether the analysis counts bytes, or packets. The first 
two parts of the specification determine which data- 
base files are read. As the files are read in, all records 
are compared against the filter and the ones not match- 
ing any rule are ignored. The records at this point rep- 
resent the traffic the analysis will be run on, but the 
database performs two more operations to help the 
analysis engine: it samples and sorts the data. 


Analyses that cover a large time interval and 
don’t specify very selective filters can read millions of 
flow records from the database. Given that the analy- 
sis engine implements complex algorithms in python, 
it runs slow on this many records. Before passing the 
results to the analysis engine, we apply the same adap- 
tive sampling algorithms used when writing the data- 
base to ensure that the number of flow records passed 
to the analysis engine is not very large (no more than 
100,000 in the current version). The database also 
sorts the records by the field used in the analysis. 


The Analysis Engine 
The current version of Netpy has an analysis 
engine implemented entirely in python.’ It runs the 


‘Netpy’s name comes from the fact that it does network 
traffic analysis in python. 
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hierarchical heavy hitter algorithms and all other algo- 
rithms doing the analysis of the traffic. Analyses can 
complete in under five seconds or take as long as a 
minute. Running a destination address hierarchy anal- 
ysis for an hour’s worth of traffic takes 5.3 sec to com- 
plete. Running a application hierarchy analysis on the 
same time interval takes 2.7 sec. On a 24 hour time 
period the analysis takes 34.7 sec and 27.0 sec, for 
address and application analysis, respectively. We plan 
to reimplement most analysis algorithms in C and we 
expect a speedup by at least a factor of 100. It happens 
quite often that the user asks for the same analysis 
again, for example by pushing the “back” button in 
the interface. To avoid accessing the database and 
doing the computations again, the analysis engine 
keeps a cache of analysis results and it reads the 
results of old analyses out of this cache. The size of 
the cache is modest because the results of the analyses 
are typically quite small. The entries in the cache are 
gzipped binary dumps of the analysis data structures, 
the average file size is | KB. 


The database and the analysis engine are nor- 
mally part of the same process as the GUI or the con- 
sole. It is also possible to run the GUI remotely and in 
this case the analysis engine runs as a daemon on the 
machine with the database. 


The Console and The Graphical User Interface 


The console and the graphical user interface, 
both implemented in python, are the two interfaces to 
Netpy. The console supports a: simple language of 
commands for updating the database and performing 
analyses. It is the only interface that allows the user to 
add new NetFlow data to the database and delete old 
flow records. The console can accept commands inter- 
actively or as a script. The GUI, built using the 
wxPython user interface toolkit, visualizes the traffic 
analysis results and helps the user navigate the traffic 
data. Drill-down through clicks on graphical elements 
representing IP address prefixes or port ranges is inte- 
grated with filters. The “‘back” and ‘“*forward”’ buttons 
further facilitate the exploration of the traffic mix. 


Sampling Algorithms Used By Netpy 


There are two measures of traffic that Netpy 
analyses can choose to compute: the number of bytes, 
and the number of packets within various categories of 
traffic that make up the traffic mix. The aim of the 
sampling algorithms is to take a large number of flow 
records and reduce it to a smaller sample that is an 
unbiased, low error representation of the original traf- 
fic. By sampling we fundamentally loose information, 
so it is unavoidable that there will be errors when we 
estimate the traffic of some categories of traffic, and 
categories with little traffic are especially vulnerable. 

Our aim is to pick a sampling function that 
ensures that the sampling error is small for the large 
categories of traffic. Let’s focus on byte counts first. A 
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simple solution is to sample each flow record with 
probability p, and for all sampled flow records to mul- 
tiply their byte counts by 1/p to compensate for the 
flow records that were not selected in the sample. For 
example if p = 1/5 we would multiply by 5 the byte 
counts of all the sampled records. This method ensures 
that the number of flow records is reduced by approxi- 
mately a factor of p. The errors introduced by this 
method are not very high if the sizes of the flows are 
close, but if there are a few very large flows the errors 
can be significant. Say the traffic mix consists of 
1,000 flows of 10 KB each and one flow of 10 MB, 
and thus the actual total traffic is 20 MB. The sample 
will contain around 200 of the small flows, each 
counted with 50 KB of traffic so their contribution 
will be estimated correctly at around 10 MB, but the 
situation is different with the large flow: if it doesn’t 
get sampled (and this has a probability of 80%), we 
don’t count it at all, if it gets sampled, we count it as 
50 MB. Thus we either underestimate the total traffic 
by a factor of (10 +0)/20 = 0.5, or we overestimate it 
by a factor of (10 + 50)/20 = 3. 


The solution to this problem is to use size depen- 
dent sampling, also known as smart sampling which 
was proposed by Duffield, et al. [4]. This methods 
picks the sampling probability in a way that favors the 
large flows. More exactly the algorithm picks a thresh- 
old z, and the flows with size s = z are kept in the sam- 
ple while the ones with s <z are kept with probability 
Ps = siz. If one of these small flows is sampled, its 
byte count is multiplied by 1/p, = z/s which gives us a 
byte count of s + 2/s =z. 

Smart sampling has the property that if the origi- 
nal set of flow records has a total traffic of 7, the 
expected number of flow records after sampling is at 
most 7/z, irrespective of how many flow records the 
original set has, and how their sizes are distributed. It 
also has the property that if the total traffic of a cate- 
gory is C the standard deviation (average error) of the 
estimate of the traffic of the category after sampling is 
at most VCz, but it can be smaller depending on the 
distribution of the sizes of the flows that are part of the 
category. For example if the total traffic is T= 100,000 
MB, and we use a threshold of z = 1 MB, the number 
of flow records in the sample is expected to be at most 
100,000 (if the original traffic mix has many flows 
significantly larger than 1 MB, the number of flow 
records in the sample will be significantly below 
100,000). If we want to estimate the total traffic 7 
based on the sample, the standard deviation of the 
result will be at most V7z = 316 MB and the the prob- 
ability that we overestimate or underestimate the total 
traffic of 100,000 MB by more than 3 V7z = 948 MB 
(an error of less than 1%) is below 0.3%. If we look at 
a smaller category with a traffic of C= 10,000 MB, 
the probability that we underestimate or overestimate 
its traffic by more than 3VCz = 300 MB (an error of 
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3%) is below 0.3%. The larger z, the smaller the sam- 
ple size, the larger the errors. If we increase z by a fac- 
tor of 100, we reduce the sample size by a factor of 
100, but we increase the errors by a factor of 10. 


When adding data to the database Netpy uses the 
limit imposed on the disk usage of the database to indi- 
rectly determine the threshold z. For example if the limit 
for one hour’s data is set to 10 MB in links.conf, this 
translates to 853 KB for each of the 12 files representing 
a five minute bin, and since the size of a flow record is 
21 bytes this translates to 41,600 records. Thus the 
value of z will be at most one 41,600th of the total traf- 
fic for the five minutes. This translates to a standard 
error for the estimate of the total traffic during those five 
minutes of at most \ 1/41, 600 = 0.49%. If we look at a 
smaller category of say 1% of the traffic during those 
five minutes the standard error of the estimate of the 
components traffic is at most 4.9%. Of course, if one 
looks at longer time periods (an hour, or a day) since 
there will be more flow records, the relative errors in the 
estimates will go down. 


The sampled flow records used for estimating 
byte counts can also be used for estimating packet 
counts. Since the sampling is biased towards flows 
with many bytes it will also catch flows with many 
packets. Packet sizes vary between 40 bytes and 1500 
bytes so it will happen that a flow with fewer larger 
packets will be preferred over a flow with more but 
smaller packets, but since the ratio between the largest 
and the smallest packet is only 37.5 the types of patho- 
logical errors that are possible with uniform record 
sampling are not possible. Let s’ be the number of 
packets in a flow record. If s = z, the flow record will 
be sampled and s and s’ will remain unchanged. If 
s <z and the flow record is sampled we multiply the 
packet count by I/p, and thus have a packet count of 
s’ + z/s in the flow record we keep in the sample. The 
errors in the estimates for the number of packets in 
various categories of traffic are similar to the errors in 
the byte counts. 


Future Work 


While Netpy is the result of a lengthy design and 
development process, we plan to improve it further. A 
first thing to do is to add flow counting functionality 
(partially implemented) because flow counts are better 
at revealing many types of traffic a security-conscious 
network administrator might want to know about such 
as scans and floods with source addresses spoofed at 
random. We can group the improvements we plan into 
improvements that will increase the speed of the analy- 
ses, and improvements that will increase their power. 

The current performance bottleneck is the hierar- 
chical heavy hitter algorithms in the analysis engine. 
We plan to reimplement all hierarchical heavy hitter 
algorithms in C and based on measurements of Auto- 
Focus’ C backend that runs similar algorithms we 
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expect a speedup by at least a factor of 100. The data- 
base read can also become a bottleneck when reading 
large amounts of data (e.g., running an analysis on an 
entire month). We plan to address this performance 
bottleneck by conceptually keeping multiple versions 
of the database a very small one with very aggressive 
sampling, a medium one and a large one with mild 
sampling. An analysis on long time intervals would 
use the coarsest database to reduce the amount of disk 
reads, while one on short time scales would use the 
most detailed one to get accurate results. In practice 
we can integrate these multiple conceptual databases 
into a single file hierarchy. More compact encodings 
of flow records or the use of gzip to compress the flow 
record files will reduce disk usage and increase perfor- 
mance since less data will have to be read. 


There are a few directions in which we plan to 
increase the power of Netpy’s analyses. Due to the 
five minutes bins used currently the analysis cannot 
look at a granularity finer than five minutes, even 
though the original NetFlow data would have allowed 
it. By adding small timestamps to the flow records we 
hope to be able to support analysis at the granularity 
of seconds. Another direction of improvement relies 
on the observation that while currently all analyses 
work on a portion of the traffic mix, it often makes 
sense to compare the current traffic against historical 
traffic to find the things that have changed. We plan to 
extend Netpy with “comparison reports” working on 
two data sets at a time. A third direction plans to 
address limitations due to the fact that the analysis of 
IP addresses relies on the implicit hierarchy in the IP 
address space. The problem with this approach is that 
because of how the IP address space is allocated, pre- 
fixes are not always meaningful, they can include por- 
tions of unrelated organizations. We plan to extend 
Netpy with three more hierarchies for IP addresses: 
one based on DNS reverse mappings of IP addresses, 
one based on whois data, and one based on BGP rout- 
ing table information. Hierarchical heavy hitter algo- 
rithms can be easily adapted to all of these hierarchies 
and they can provide valuable new insights into the 
traffic mixes on our networks. 


Our hope is that many of these improvements 
will be implemented and mature enough for wide- 
spread use by the end of the 2005. 


Conclusions 


IP traffic can be unpredictable and traffic analy- 
sis can help with incident response as well as with 
long term planning of network growth. This paper 
presents Wisconsin Netpy, an application for analyz- 
ing and visualizing NetFlow traffic data. While Netpy 
is certainly not the first application in this space, we 
believe that it incorporates important new ideas that 
enable powerful exploratory analyses of the traffic mix 
not supported by other tools currently used by network 
administrators. The use of a small database of sampled 
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traffic enables prompt analyses that allow the network 
administrators to refine their queries iteratively. The 
unidimensional and the novel bidimensional hierarchi- 
cal heavy hitter analyses can provide a detailed view 
of the traffic. The visualization of analysis results in 
the form of time series plots and heatmaps makes it 
easier for a human observer to absorb information 
about the composition of the traffic mix. We hope that 
Netpy will contribute to a better understanding of how 
networks are used and through this understanding to 
better managed networks that offer more reliable ser- 
vice to millions of Internet users worldwide. 
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ABSTRACT 


The frequent and large-scale network attacks have led to an increased need for developing 
techniques for analyzing network traffic. If efficient analysis tools were available, it could become 
possible to detect the attacks, anomalies and to appropriately take action to contain the attacks 
before they have had time to propagate across the network. This paper describes NetViewer, a 
network monitoring tool that can simultaneously detect, identify and visualize attacks and 
anomalous traffic in real-time by passively monitoring packet headers. NetViewer represents the 
traffic data as images, enabling the application of image/video processing techniques for the 
analysis of network traffic. 


NetViewer is released free to the general public. By employing a freely available 
visualization tool, the users of Net Viewer can comprehend the characteristics of the network traffic 
observed in the aggregate. NetViewer can be employed to detect and identify network anomalies 
such as DoS/DDoS attacks, worms and flash crowds. NetViewer can also provide information on 
traffic distributions over IP address/port number domains, utilization of link capacity and 


effectiveness of Quality of Service policies. 


Introduction’ 


The frequent and increasing malicious attacks on 
network infrastructure, using various forms of denial of 
service attacks, have led to an increased need for devel- 
oping techniques for analyzing network traffic. If effi- 
cient analysis tools were available, it could become 
possible (i) to detect the attacks, anomalies and (11) to 
appropriately take action to mitigate the attacks before 
they have had time to propagate across the network or 
to cripple the infrastructure. These tools may be in turn 
useful for traffic engineering purpose since the network 
traffic analysis provided by these tools could lead to the 
identification of resource bottlenecks and peak usage. 


A variety of tools for flow-based measurement 
have arisen from both the commercial and free soft- 
ware communities. To study and classify traffic on the 
network based on usage and protocols, a number of 
tools such as FlowScan [PLOO00], Cisco’s FlowAna- 
lyzer, and AutoFocus [ESTsv03], are used as traffic 
analyzers. While flow-based features within the net- 
work infrastructures are convenient, such approaches 
may not be sufficient for reliable and fast application. 
Some of these tools provide real-time reporting capa- 
bility, but much of the analysis is done off-line. These 
tools have been effectively utilized for traffic engi- 
neering and postmortem anomaly detection. 


However, rigorous real-time analysis is needed 
for detecting and identifying the anomalies so that 
mitigation action can be taken as promptly as possible. 


This work is supported by NSF grants ANI-0087372, 
0223785, Texas Higher Education Board, Texas Information 
Technology and Telecommunications Taskforce and Intel 
Corp. 


Some of these tools are based on the volume of traffic 
such as byte counts and packet counts. When links are 
not sufficiently provisioned, normal traffic volumes 
may reach the capacity of the links most of the time. 
In such cases, attack traffic may not induce significant 
overshoot in traffic volume (merely replacing existing 
normal traffic) and hence may make traffic volume 
signal ineffective in detecting attacks. 


Sophisticated low-rate attacks [EUZk03] and 
replacement attacks, which don’t give rise to notice- 
able variance in traffic volume, could go undetected 
when only traffic volume is considered. Furthermore, 
the tools which collect and process flow data may not 
scale to high-speed links as they focus on individual 
flow behavior. Our tool tries to look at aggregate 
packet header data in order to improve scalability. 


Intrusion detection systems (IDS) such as Snort 
and Bro are an important part of network security 
architecture and signature database-based monitoring 
of network traffic for predefined suspicious activity or 
patterns. These tools are widely deployed by network 
administrators. This detection principle relies on the 
availability of established rules of the anomalous or 
suspicious network traffic activity. While the identifi- 
cation mechanism of the IDS tools provides fine-grain 
control of network flows, they however need to be 
updated continuously with the latest rules for coping 
with novel attacks. Our approach tries to develop a 
generic mechanism independent of specific anomalies 
for improving adaptability. 

In this paper, we describe a tool named 
NetViewer for traffic anomaly detection based on 
analyzing the distribution of traffic header data, in 
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postmortem and in real-time. We adopt a network 
measurement-based approach that can simultaneously 
detect, identify and visualize attacks and anomalous 
traffic in real-time. We propose to represent samples 
of network packet header data as frames or images. 


With such a formulation, a series of samples can 
be seen as a sequence of frames or video. This enables 
techniques from image processing and video compres- 
sion such as scene change analysis and motion predic- 
tion to be applied to the packet header data to reveal 
interesting properties of traffic. Our work here brings 
techniques from image processing and video analysis to 
visualization and real-time analysis of traffic patterns. 


We show that “scene change analysis” can 
reveal sudden changes in traffic behavior or anomalies 
[LELs03, ZHAks93, LIEke97, SHEd95, GYAkc03]. 
We show that “motion prediction” techniques can be 
employed to understand the future patterns of some of 
the attacks. We show that it may be feasible to repre- 
sent multiple pieces of data as different colors of an 
image enabling a uniform treatment of multidimen- 
sional packet header data. NetViewer can give an intu- 
itive and descriptive illustration of network traffic, 
with visible features, to network operators. 


Internet A.D. 
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Figure 1: The installation locations of NetViewer. For 
ingress/egress filtering, it can be employed in a 
router or at the boundary of an administrative 
domain. 


Environment 


The visibility of fields in the network packets 
may be impacted by the location where traffic is 
observed. NetViewer can be applied for analyzing 
inbound/outbound traffic at administrative domain 
(AD) edge router as shown in Figure 1. The AD repre- 
sents the access link, ISP (Internet Service Provider) 
and intra-domain IP-based networks such as enterprise 
networks and campus networks. 


Employing ingress filtering with Net Viewer moni- 
tors the flow of traffic as it enters a network under 
administrative control. NetViewer can be combined with 
a Quality of Service (QoS) policy framework to rate 
limit high-bandwidth flows identified by NetViewer. 


Egress filtering with NetViewer inspects the flow 
of traffic as it leaves a network under administrative 
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control. There are typically policy regulations for 
inner machines initiating outgoing connections to the 
Internet. Outbound filtering has been advocated for 
limiting the possibility of address spoofing, i.e., to 
make sure that source addresses correspond to the des- 
ignated addresses for the AD. Traffic monitoring at a 
source network enables a detector to detect attacks 
early, to control hijacking of AD machines, and to 
limit the liability from such attacks and the squander- 
ing of resources. NetViewer can help prevent compro- 
mised systems on a network from attacking systems 
elsewhere through egress filtering. 


Our traffic attack/anomaly filtering efforts have 
been targeted at the edge environment due to resource 
utilization issues and source address spoofing concerns. 


Goals 


NetViewer’s goal is not to detect and eliminate 
the anomalies eradicatively, but to bring the anoma- 
lous traffic under control in real-time so that the miti- 
gation mechanisms could be deployed fast to counter 
the threats of the anomalies. We also want NetViewer 
to not contain any legitimate traffic in the process of 
containing attack traffic. Our goal is to detect and con- 
tain 99.7% of the attack and anomalous traffic at a 
false alarm rate of 0.3%. Given the current and grow- 
ing network link speeds, anything we implement 
needs to be fast and efficient enough to not exces- 
sively burden existing network infrastructure. 


As an operational goal, we want that the contain- 
ment is centrally processed at a router level, rather 
than something that the end users would have to deal 
with. Netviewer can work with packets on the fly at a 
router in real-time mode and packet traces in libpcap 
or NetFlow format in post-mortem mode. 


Architecture 


NetViewer’s architecture is mainly focused on 
performance, simplicity, and versatility. NetViewer 
system’s architecture consists of five major software 
components: the packet parser, the signal computing 
engine, the detection engine, the visualization engine 
and the alerting engine, which are programmed in 
ANSI C and Matlab language [MATO1]. 


The Packet Parser 


The packet parser engine is responsible for col- 
lecting and processing raw packets and traffic data 
exported from routers. NetViewer can work with traf- 
fic records in postmortem or work with more aggre- 
gate data upon packet arrival in real-time. It can parse 
packets on the fly and parse network packet header 
traces with libpcap (packet capture library) [PCAP94], 
Cisco’s NetFlow [NETF] and NLANR formats 
[NLA02] such as DAG, Coral, and TSH (time 
sequenced headers). 


Traffic volume, such as packet counts, byte 
counts and the number of flows, can be used as a 
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signal, and fields in the packet header, such as 
addresses, port numbers and protocols, can be 
employed as an observed domain. According to opera- 
tor’s concern, NetViewer then generates images of the 
distribution of traffic intensity in the chosen domain. 
Based on the kinds of traffic data and the header 
domain, we categorize the image-based signals into 
address-based, flow-based and port-based_ signals. 
Address-based signal employs packet count distribu- 
tion over address domain (either source address alone, 
or destination address alone, or a 2-dimensional 
source and destination address domain). Flow-based 
signal employs the flow number distribution over 
address domain(s). Port-based signal employs packet 
count distribution over port number domain. 


Traffic headers such as addresses and port num- 
bers have larger spaces over which data is distributed, 
that is, 2°? IPv4 addresses and 2'° port numbers. Any 
developed technique should be simple enough to be 
deployable, i.e., should not be expensive in terms of 
memory and processing resources. In order to address 
the problem of large domain spaces, we have employed 
a concise data structure for reducing the domain space 
[KIMrv04]. We explain this data structure using 
address-based signal as an example. This data structure 
count{i][/][t] represents the data sample at time f. 


The data structure consists of four arrays count[4] 
for the 4 bytes of the IP address. Within each array, we 
have 256 locations, for a total of 4*256 locations = 1024 
locations. By default, the size of one location is set to 16 
bits. A location count{i][/][4] is used to record the packet 
count for the address j in ith field of the IP address in 
time interval t. This provides a concise description of the 
address instead of 2°” locations that would be required to 
store the address occurrence uniquely. Upon packet 
arrival, the corresponding four positions of the data 
structure are updated through scaling. 

The Signal Computing Engine 

Each sampling period, the packet counts of the 
entire traffic are recoded to the corresponding positions 
of each IP address byte-segment, and the normalized 
packet count is quantized and represented using Equa- 
tion (1). Each resultant normalized packet count then 
represents the intensity of the corresponding pixel in the 
image representation of the traffic as shown in Figure 2. 

— count{iVI4 i=0,1,2,3 (1) 
ye 255 "= 
¥ count{i|¥td as aos 
j-0 

In order to quantitatively analyze the network 
traffic anomalies, we compute correlation and deltas 
based on normalized packet counts. Consider two 
adjacent sampling instants. We can define correlation 
signal at sampling point ¢ by (2-1), which measures 
correlation of traffic intensity at a particular address. 
Delta is defined as the difference of normalized packet 
counts by (2-2), which is a useful signal at the begin- 
ning and ending of attacks. 
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Cyje = Pit * Pijt-1 (2-1) 

AD ijt = Pit — Pijt-1 (2-2) 

We employ the variance of pixel intensities in the 

image as traffic signal for scene change analysis, 

which is denoted by S,. Using the variance of these 

image signals for deriving thresholds, we can obtain 

an approximation of the energy distribution of the nor- 

malized packet counts within the observation domain 
as follows: 






3 255 
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where pj;, are pixel intensities and 
3.255 
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Upon each sampling instant, the aggregate traffic 
signal is instantaneously calculated based on accumu- 
lated data structure for real-time analysis. 


The Detection Engine 


The detection engine employs a theoretical basis 
for deriving thresholds for analyzing traffic signals and 
anomaly detection. For 30-based statistical analysis, 
we set two kinds of thresholds, a high threshold 7}, and 
a low threshold 7,;. When we respectively set the 7), 
and 7, thresholds to +30 of aforementioned traffic sig- 
nal distributions in ambient traffic, attacks can be 
detected with an error rate of 0.3% (if the signal is nor- 
mally distributed) which can be expected as target false 
alarm rate as (4-1) [NISOS5]. For deriving initial thresh- 
olds from background traffic, a tune-up procedure is 
necessary just after powering up. By default, it is set to 
120 samples; that is 2 hours in case of | minute sam- 
ples. To analyze the statistical properties of normal traf- 
fic dynamically, we employed an exponential weighted 
moving average (EWMA) of normal traffic free of 
attacks. The dynamic average of the traffic is updated 
at every sampling point excluding attack periods. 





The detection engine can judge the current traffic 
status by calculating the standard intensity deviation 
of signals in each sampling instant by (4-2). The ana- 
lyzed information will be compared with historical 
thresholds of traffic to see whether the traffic’s charac- 
teristics are out of regular norms. Sudden changes 
over 30 in the analyzed signal are expected to indicate 
anomalies. 

X~N(u, 67) > Pr(ui-36 < XS +30) = 99.7% (4-1) 
— normal, if T, < Sg < Ty 2 
Be Ee attack, if Ss <T;, or Ty < So (4-2) 

Also, with mean, standard deviation and signal 
value at every sampling point, the detection engine 
would compute the probability of anomaly assuming 
the normal distribution [KILn02, NISOS5]. 


The Visualization Engine 
The visualization engine employs graphic library 
of Matlab (in this implementation) for displaying 
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traffic signals and images. The visualization engine 
produces user-friendly, images of network traffic. As 
shown in Figure 2, the visual parts of NetViewer’s 
main screen make up four primary components: the 
general traffic profile, traffic distribution signals, traf- 
fic images and anomaly reports. While the profiling 
and reporting components are expressed in text, the 
traffic signals and images are visualized in graphs. 
The visualization engine enables NetViewer to offer 
these visual measurements as a real-time motion pic- 
ture. It could help the network operators recognize the 
trends and transitions in network traffic. 


The visualization engine plots the standard devi- 
ation of traffic intensity computed by (3) versus the 
sampling instant in source and destination domains 
respectively as network traffic signals. If the anomaly 
is detected, a red dot is marked on the bottom and 
calls the network operator’s attention. NetViewer’s 
viewing window is controllable for short-term and 
long-term analysis purposes. By default, the window 
is 60 sampling points, which is the latest | hour with | 
minute samples. 


Each element of the data structure computed by 
(1) corresponds to a rectangular area (or pixel) in the 
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traffic image. The values of the elements of the data 
structure are indexed into the current color-table that 
determines the color of each pixel. The color of each 
pixel shows the intensity of traffic at the source or des- 
tination or (source, destination) pair in 2-dimensions. 


The descending order of intensity is black, red, 
orange, yellow and white. Each quadrant corresponds 
to each byte in IP address structure. In 1-dimensional 
source or destination domain, each quadrant consists 
of 16 by 16 pixels for mapping 256 elements of one 
byte of IP address as shown in Figure 3(a). Each quad- 
rant maps the 0 to 255 values of one byte of IP address 
in a row-major order. Thus four quadrants consist of 
the entire 4-byte IP address. The four bytes 0 to 3 of IP 
address are also organized as quadrants in a row-major 
order. In 2-dimensional image, the x-axis in each quad- 
rant corresponds to the distribution of the destination 
IP addresses, and y-axis that of the source addresses as 
shown in Figure 3(b). In each quadrant, source and 
destination addresses consist of 256 by 256 pixels. 


The Alerting Engine 


Once anomalies are detected through scene 
change analysis, the alerting engine scrutinizes the 


Signal: o of Packet Count 
Time . [7-26-2003 6:8:0 ~ 6:9:0 ] 
Sampling: 120sec, DB: 16bit 


aeariats SW - 20470.835 Kbps 


BY: 3.213 Kpps 

31 TCP:UDP:ICMP: ete = 97.5: 1.321.002 

*Top 5 flows / 17812 flows Ptl P(%) BY(kbps' 
(1] 210.116.148.155 (01951) -> 150.254.015.130 (02120) [006] 044 01676.3 
{2] 134.075 007.041 (38740) -> 128.230 129 106 (00119) [006) 043 016506 
(3] 134.075.007.041 (88626) -> 128.230.129.106 (00119) [006] 03.7 014425 
[4] 134.075.007.041 (68694) -> 018.181.000.026 (00119) 006} 03.7 01427.4 
{5} 210.219.033.042 (34661) -> 131.142.042.025 (00873) {006} 03.5 000016 


Source 





/ Destination (X} IP 


(0) 134, C=21.57% P=21.51% A=-0.11% (0) 128, C=07.55% P=07.28% A=-0.54% { Identified Suspicious Source IP address = 3] 
(0) 155, 0=53.59% P=52.93% A=-1.33% 0} 131, C=49.40% P=48.67% A=-1.48% 134.075 xxx.xxx C=21.57% P=21.51% A=-0.11% 
fO}210, C=07.49% P=08,08% A=1.13% [1] 183,  C=47.47% P=45.57% A=-1.90% 155.230.014.xxx C=53.59% P=52.93% A=-1.33% 


(t] O75,  C=21.57% P=21.51% A=0.11% 
[1] 110,  C=05.02% P=05.30% A=0.58% 
[1] 230,  C=53.61% P=52.94% A=-1.33% 
[2] O07, C=18.36% P=18.48% A=0.24% 
(2] 014, C=49.00% P=48.13% A=-1.76% 
[3 081, C=17 77% P=17 78% A=0.013% 
GB] 050, C=47 65% P=45 81% A=-1. 68% 


210.xxx.xxx,xxx C=07.49% P=08,08% A=1.13% 


| Identified Suspicious Destination IP address = 2 } 
128. xxx.xxx.xxx C=07 55% P=07.28% A=-0.54% 
131.183.xxx.xxx C=49.40% P=43.67% A=1.49% 


Figure 2: A running example of NetViewer. 
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above quantities by (1) and (2) for identification pur- 
poses. From the predefined correlation and delta 
thresholds, the alerting engine can identify the IP 
addresses of suspicious attackers and victims. Based 
on the revealed IP addresses, we closely investigate 
each address on the basis of statistical measurements. 
This inspection will lead to some form of a detection 
signal that could be used to alert the network admin- 
istrator of the potential anomalies in the network traf- 
fic. The alerting engine generates the detection reports 
in online image as well as in an offline file. 


NetViewer’s Functionality 


NetViewer and its component engines are 
responsible for profiling and monitoring raw packets 
or trace data exported from routers. Net Viewer moni- 
tors each packet and maintains data structures based 
on the observed domain (address, port numbers etc.). 
When anomalies are detected, NetViewer reports its 
detection results and may optionally take containment 
actions. It may be configured to either archive for 
postmortem analysis or discard the counter contents 
after processing. 


Controllable parameters, such as window size for 
determining the amount of retained data and thresholds, 
can be configured before or after the packet parser 
engine takes an action. By default, a sampling interval is 
set to 60 seconds for deriving most stable traffic images, 
and a sampling duration ratio is 1:1. That is to say, we 
sampled for 30 seconds and paused for 30 seconds for 
reducing the processing requirements. Our techniques 
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(a) 1 dimension 


NetViewer: A Network Traffic Visualization and Analysis Tool 


are light-weight enough for traffic to be continuously 
sampled without any pause periods. 


Figure 2 is a sample NetViewer graph of real 
traffic with attack at an access link. 


Traffic Profiling Function 


The upper right text in Figure 2 shows the gen- 
eral information of current network traffic. The profile 
includes the selected signal type, the local time, the 
selected sampling period, number of bits used to rep- 
resent traffic intensity of individual pixels in the data 
structure, and the bandwidth in Kbps (bits per second) 
and Kpps (packets per second). 


The next line illustrates the proportion occupied 
by each traffic protocol. It is possible to determine 
whether the current traffic is behaving normally 
through correlating it to that of previous states of traf- 
fic from a protocol viewpoint. This is based on the 
observation that during the attacks, the protocol 
employed by the attack traffic should see considerably 
more traffic than during normal traffic. 


The Top 5 flows term shows the topmost five flows 
out of the total flows in the selected traffic volume, 
which can be packet count, byte number or the number 
of flows. It is expressed as a quintuple of source IP 
address/port number and destination IP address/port 
number, exploited protocol, occupied proportion in per- 
centage, and bandwidth in Kbps. It is implemented using 
LRU (Least Recently Used) policy with partial state for 
identifying long-term high-rate flows [SMIkr01]. 
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(b) 2 dimension 


Figure 3: The visualization of the network traffic signal in the IP address domain. 
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The profiling information presented by Net- 
Viewer assists in understanding the general nature of 
the traffic at the monitoring point. 


Monitoring Function 


The two upper left sub-pictures in Figure 2 illus- 
trate traffic distribution signal over the latest prede- 
fined (and adjustable) time-window. These two pic- 
tures map to source domain and destination domain 
respectively. The captions above each picture explain 
the current traffic distribution. The Source IP term 
means that this signal is originated from packet counts 
in the source IP address domain. Based on the opera- 
tor’s selection, this term can be changed to Source 
FLOW which analyzes the number of flows over the 
source IP address domain, or to Source PORT which 
analyzes packet counts in the source port domain, or 
to Source MULTIDIMENSIONAL which analyzes multiple 
components of different distributions in the source IP 
address domain. 


The Pr term estimates the anomalous probability of 
current traffic distribution assuming Gaussian distribution. 
The probability is computed from the normal distribution 
table based on the traffic signal, its u and o [NISO5]. We 
can be informed the possibility of anomalous traffic quan- 
titatively. The Signal term is computed by (3). The “L” 
and “o” terms mean the mean value and the standard 
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deviation of distribution signal using EWMA. The left 
two figures in y-axis and dotted vertical lines illustrate the 
+30 levels respectively. The above statistical measure- 
ments are dynamically updated at every sampling point 
excluding attack periods. 


The red dots located on the bottom of the each 
sub-picture are marked when 30-based statistical anal- 
ysis detects anomalies. The detection signal automati- 
cally triggers to identify the IP addresses of sources 
and destinations and can be used to alert traffic anom- 
alies to the network operator. 


Anomaly Reporting Function 


The center three sub-pictures in Figure 2 illus- 
trate image-based traffic in the source/destination IP 
address domain and the 2-dimensional domain. 


The color and darkness of each pixel indicate the 
intensity of traffic of corresponding IP address. In case 
of normal traffic, the aggregate traffic does not form 
any regular shape due to dispersibility of traffic of vari- 
ous and numerous flows in time and space. In the case 
of abnormal traffic, however, the traffic pattern of net- 
work may change and these changes could be exhibited 
in the visual images. From destination IP address in 
Figure 2, a specific area of IP byte2 is shown in a 
darker yellow shade. It illustrates that the current traffic 
is concentrated on a (aggregated) single destination or a 


2 Frame = $8 .Source/Destination iP 
IP bye 0 P 


IP byte 4 P byte 1 
15 0 255 
| a 
| 
| 
we 
2 
8 
| ny D4 
| &§ 3 
S oe 
| & 
ry 
— = g 
} ow 
i uo 
i oo 
iP byte 2 IP byte 3 


destination IP address 


[Worm propagation type of attack traffic] 


id) Frame = 1659 Source IP 


{e) Frame = 1659 Destination IP 5 crama = 1689 Sourco/Destnation IP 


IP byte 0 IP byte 1 
0 2550 255 
° 


ossz 
SSAIPPR GI so/N08 


ssc 





IPoyeO IP bytet IPbyeQ IPbyte4 
9 a en ' pied 45 

} wa | 

a om) 

=| s oe 

nN nN 

lem gi b 

ws 1 vil 

Q| Qt 

°| s ‘ oe} 

=| a ‘ 

| 

| 

| 

ot. . Jj 3 . 
IPhye2? IPoye3 IPbyte2 IPbye3 


IP tye 2 IP byte 3 
destination IP address 


[Distributed Denial of Service (DDoS) type of attack traffic] 


Figure 4: Various attack images reported by Net Viewer. 
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subnet. NetViewer also shows that a specific source, Additionally the detection report is optionally 

i.e., an attacker, monopolizes network traffic, shown in generated in a file format. The upper part of Figure 5 

the form of a line in the 2-dimensional domain. shows the identified addresses responsible for the 
The exposed images can show various kinds of anomalies over the four byte-segment levels and the 

patterns according to the nature of attacks. Figure 4 lower part of Figure 5 illustrates source and destina- 

shows the visual expression of a worm and a DDoS tion addresses of attacks. 

attack respectively. While the horizontal lines imply that Auxiliary Function 


the same source targets multiple destinations in a worm 
attack, the vertical lines reveal that several machines (in 
a subnet) are targeting a single server in a DDoS attack. 


NetViewer could support various auxiliary func- 
tions. The video/frame representation of network traffic 
i oan enables motion estimation based techniques for attack 

Once anomalies are detected, the identified IP prediction. Unassigned address ranges can be clearly 
addresses of sources and destinations are revealed in marked in the generated images to identify address 


byte-segment level and 4-byte whole structure simulta- spoofing. We describe some of these functions here. 
neously as shown in the lower text of Figure 2. These 


identified IP addresses are quantitatively investigated Multidimensional Image 


on the basis of statistical measurements using the corre- Up to now, we have focused on address-based 
lation (C), the possession ratio (P), the delta (A) image on which normalized packet counts are visualized 
between consecutive frames and the black list (S) com- in the address domain. By the operator’s selection, 
puted by the signal computing engine. S recorded in the NetViewer’s main screen can be changed to flow-based 
last column indicates black listing which is successively image which visualizes the number of flows over the 
identified and refined over recent sampling instances. It address domain, or to port-based image which displays 
could help network operators make a final decision. packet counts in the port domain, or to multidimensional 
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Source IP[O] 134. correlation = 17.48% possession= 18.77% delta= 2.50% S 
Source IP[0] 141. correlation= 4.33% possession= 3.94% delta= 0.79% S 
Source IP[0] 155. correlation = 58.20% possession = 56.80% delta= 2.84% S 
Source IP[O] 210. correlation= 5.66% possession= 6.51% delta= 1.60% S 
Source IP[1] 75. correlation = 17.47% possession= 18.77% delta= 2.51% S 
Source IP[1] 110. correlation= 4.62% possession= 5.25% delta= 1.21% S 
Source IP[1] 223: correlation= 4.31% possession= 3.94% delta= 0.78% S 
Source IP[1] 230. correlation = 58.21% possession = 56.84% delta= 2.76% S 
Source IP[2] 7. correlation= 15.59% possession= 17.02% delta= 2.74% S 
Source IP[2] 14. correlation = 53.99% possession = 52.31% delta= 3.41% S 
Source IP[3] 41 correlation= 15.16% possession= 16.36% delta= 2.30% S 
Source IP[3] 50 correlation = 52.58% possession = 50.83% delta= 3.54% S 
Identified No. Ist = 4, 2nd = 4, 3rd = 2, 4th = 2 
Destination IP[0] 18. correlation= 4.37% possession= 3.88% delta= 1.01% S 
Destination IP[0] 128. correlation= 6.08% possession= 7.01% delta= 1.75% S 
Destination IP[0] 131. correlation = 53.65% possession = 52.33% delta= 2.67% S 
Destination IP[1] 181. — correlation= 56.03% possession= 54.00% delta= 4.15% S 
Destination IP[3] 26 correlation= 3.89% possession= 3.58% delta= 0.65% S 
Identified No. Ist = 3, 2nd = 1, 3rd = 0, 4th = | 
* Identified Suspicious Source IP address(es) 
134. 75. 7. 41 correlation= 17.48% possession= 18.77% delta= 2.50% S 
141.223.xxx.xxx correlation= 4.33% possession= 3.94% delta= 0.79% S 
155.230. 14.50 correlation = 58.20% possession= 56.80% delta= 2.84% S 
210.xxX.XXxX.xxx correlation= 5.66% possession= 6.51% delta= 1.60% S 


* Identified Suspicious Destination IP address(es) 
18.XXxX.Xxx.xxx correlation= 4.37% possession= 3.88% delta= 1.01% 
128.XXX.XXx.xxx correlation= 6.08% possession= 7.01% delta= 1.75% S 


131.181.xxx.xxx correlation = 53.65% possession= 52.33% delta= 2.67% 
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Figure 5: The detection report of anomaly identification (with anonymized IP addresses). 
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image which simultaneously exhibits multiple compo- 
nents of different images in the address domain. 


In multidimensional images, individual compo- 
nents, such as packet counts, the number of flows and 
the correlation of the packet counts, can be repre- 
sented as different components (for example, Y, U, V) 
of an image or different primary colors (R, G, B). 
With the three distinct traffic components, NetViewer 
can comprehensively analyze the traffic properties of 
each IP address from diverse viewpoints. 


iP byteO Frame =25, DestinationIP IP byte 1 
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Starting position: 49 --> 144 [ +95 ]; Area size: 45 -> 46 [+1] 
Figure 6: Potential attack estimation in red using 
motion prediction. 


Attack Tracking 


During some attacks, a concentrated attack is cir- 
culated on the address space. Using motion prediction, 
it is possible to expect or anticipate the next set of tar- 
get addresses in such attacks. NetViewer estimates the 
locations of the next attack using modified motion 
prediction scheme. The results from such an analysis 
on a predictable attack are shown in Figure 6. Figure 6 
shows the ongoing attacked parts in black and the 
result of motion prediction (in red pixels) indicating 
the next set of addresses that may be a target of this 
attack. The next potential attack ranges are estimated 
based on the starting positions of current attack and 
the motion vector length. The attack tracking would be 
displayed on NetViewer’s subsidiary screen. 


Automatic Spoofed Address Masking 


In bandwidth consumption attacks such as tradi- 
tional flood-based attacks, the source IP address of the 
packet could be usually spoofed through the abuse of raw 
sockets function or the like. The deliberate falsification is 
often forged in random order or in a dictionary order. 
Random destination addresses may also be employed dur- 
ing probing for possible infiltration. SQL Slammer was 
the representative random propagation worm [CERO3]. 
And the portscan-based attacks heavily exploit the desti- 
nation port numbers in random or sequential order. 


Global address allocations are organized by types, 
which are classified according to prefixes. Currently, 
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many portions of the IP address space are still unas- 
signed by IANA (Internet Assigned Numbers Authority) 
[IPV04]. Especially, the unallocated address space can 
be clearly identified in the first byte image in NetViewer. 
Moreover, if NetViewer is located at the boundary of the 
AD, the system administrator can make sure that 
addresses correspond to the locally designated addresses 
for the AD. The packets coming into and going out of 
the AD should have the destination address and the 
source address assigned to the AD, respectively. The 
masking can be also applied to unavailable port services 
against portscan attacks if the network operator selects 
the NetViewer’s port-based functions. 
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Figure 7: Spoofed source IP addresses. 


Figure 7 shows the portion of NetViewer, where 
the blue-colored polygons indicate the reserved IP 
address space by IANA at the time of this writing. In 
legitimate traffic, there should be no pixels matching 
the unassigned space as shown in the destination IP 
image. Since packets can not be allowed to originate 
from or be destined to these spaces, the (colored) pix- 
els matching the unallocated space have to be from 
spoofed IP addresses as shown in the source IP image. 
This attack is an instantaneous attack with (randomly) 
spoofed source addresses which aimed at a specific 
machine with 160 byte-sized packets. 


The spoofed address masking can be expanded to 
IP bytes 1, 2 and 3 as well because the system operator 
knows the AD’s internal usable address range (for stub 
networks). This also enables detection of sequential 
portscan attacks through the identification of addresses 
that do not provide service at the designated ports. 


How Is NetViewer Different From IDS? 


Network intrusion detection system (NIDS) is an 
important part of network security area and signature- 
based detection approach is being widely employed by 
network operators. The rule-based matching mecha- 
nism requires that the completed analysis of attack 
patterns and the availability of established remedies be 
available beforehand. To cope with new attacks, IDS 
tools are continuously required to be updated with the 
latest rules. Currently there are a few available free- 
ware/shareware and commercial IDS tools. 


We review Snort as representative IDS [ROE99], 
[SNO], and compare the properties of Snort and 
NetViewer. We perform this comparison by running 
the systems on a live, production network. We report 
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ABSTRACT 


A] is a Java-based spreadsheet environment that enables system administrators to build small 
tools that simplify and automate common tasks, integrating real-time data across heterogeneous 
systems. Al spreadsheets can be saved to a central repository, where they are published and shared 
as interactive web portlets. In this paper, we discuss the need for administrators to create their own 
tools, how the Al environment is designed to support this need, and how A1l’s support for web 
publishing — without requiring special web programming — can enable teams to share, modify, and 
improve their tools. We also discuss the design and implementation of Al, and show a number of 


sample spreadsheets for various administration tasks. 


Introduction 


System administration frequently involves the 
development of custom scripts to manage the unique 
requirements of each site. Typically, these scripts pro- 
vide functionality lacking in commercial software 
tools because vendors cannot anticipate every task for 
every possible configuration in their tools. Custom- 
built scripts range from a few lines of code developed 
by a single administrator to speed a commonly exe- 
cuted task, to significant software systems developed 
by a team of people over several years. 


Collaboration is a common aspect of system admin- 
istration work. At most sites, expertise is distributed 
across teams who work together to design, implement, 
and maintain systems, and to troubleshoot problems. Sys- 
tem administrators often share scripts as they collaborate 
on a task, or they pass scripts to others working on simi- 
lar tasks. Scripts are often modified, customized, and 
improved, as they pass from person to person. 


In the course of system administration field stud- 
ies, we have observed several difficulties in existing 
scripting practices. Coordinated script use between 
different administrators can be hindered by a lack of 
central script repositories, inconsistent script versions, 
and varying execution environments. Web-based 
scripts provide consistent access and execution, but 
are difficult to develop. Many administrators lack the 
skills needed to field web applications, such as cgi 
scripting or J2EE servlet programming. Finally, we 
have seen many cases where the complexity of com- 
mon scripting languages (e.g., Perl, Python, and shell 
scripts) prevented administrators from modifying and 
reusing existing scripts for their own purposes. 

Though sometimes these issues result from lim- 
ited programming experience, more frequently script 
sharing problems are due to insufficient documenta- 
tion, hidden assumptions and hard-coded constants. 


All too often administrators who need a certain tool 
“re-invent the wheel,” rewriting it themselves instead 
of building on existing scripts. 


We developed Al to enable system administra- 
tors to quickly develop custom web-based tools and to 
more easily share, reuse, and improve tools collabora- 
tively. Al is a spreadsheet-like environment that 
allows system administrators to invoke existing scripts 
or access remote systems via standard protocols, such 
as SSH, JMX and SNMP. A1 includes a task-specific 
scripting language so that connections to servers can 
be opened in a spreadsheet cell, and server actions can 
be triggered by changes to cell values. Sysadmins can 
save spreadsheets to a shared repository and then exe- 
cute them from a web browser as interactive web 
portlets. Al also provides a plug-in architecture to add 
support for new components. 


A] is not meant to replace existing scripting lan- 
guages, such as Perl and Python. The power and flexi- 
bility of these languages will continue to be important 
for advanced users and experienced programmers cre- 
ating complex tools. Rather, Al is intended to provide 
an environment for a broader population of system 
administrators to create small tools that can be quickly 
and easily deployed and shared via the web. Al 
spreadsheets can be also be used as a control layer to 
execute and manage the output of existing scripts 
developed in other languages. In summary, system 
administrators using Al can build and share spread- 
sheets to access remote, heterogeneous systems, 
gather and integrate real-time data, and control various 
systems uniformly through a web-interface. 


Background 


Over the past three years, we conducted a series 
of field studies of system administration work. The 
study sites included large enterprise, university, and 
government research environments. We made fourteen 
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visits to six different sites, examining web hosting, 
database, operating system, security, and storage 
administration. We interviewed and surveyed system 
administrators about their work. We recorded a total of 
50 days of video following normal, day-to-day activi- 
ties. The results are primarily qualitative, providing a 
picture of typical work practices, tools, and problems 
faced by system administrators. 


In these field studies, we made the following 
observations relevant to administrators’ use of script- 
ing and tools: 

1) Collaboration is important for system administra- 
tors. Because no single individual can be an 
expert in every aspect of a large installation, 
responsibility is typically spread across people 
and organizations. During complex configuration 
or troubleshooting, collaboration is essential to a 
successful resolution. For example, in a detailed 
analysis of one lengthy troubleshooting session, 
we found that 90% of one system administrator’s 
time was spent communicating with others, and 
that misunderstandings greatly delayed finding 
the solution [1]. In such an environment, there is 
great potential for improving effectiveness of 
system administrators by providing tools that 
allow them to better collaborate, communicate 
system status, and share control. 

2) System administrators frequently create small 
custom tools to accomplish specific tasks. 
Administration tasks are complicated given the 
heterogeneous nature of most systems, with 
many components from different vendors and 
distinct local requirements. It is therefore not 
surprising that many tasks are not directly sup- 
ported by vendor-supplied tools. We observed 
system administrators creating small tools on- 
the-fly to solve problems and collect informa- 
tion. Only one site we observed had a script 
repository, though the procedures regulating 
script publishing limited its use. 

3) The programming abilities of system administra- 
tors vary tremendously. The system administra- 
tors we observed were very familiar with shell 
scripting, but general software and web develop- 
ment skills were usually not part of their job 
requirements. We observed the majority of scripts 
created by gurus. Novices would use scripts writ- 
ten by others, but they often did not feel comfort- 
able modifying the scripts or creating their own. 
Other administrators between the novices and 
gurus would develop scripts occasionally, but 
often had trouble understanding and modifying 
scripts written by others. Administration work 
could be aided by easier-to-use and reuse script- 
ing environments. 

4) Administrators frequently integrate information 
gathered from different tools and different sys- 
tems. This is especially true for heterogeneous 
systems containing components from different 
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vendors in which each component requires a sep- 
arate tool for access or configuration. Adminis- 
trating heterogeneous environments could be 
improved by meta-tools that access and integrate 
information from many sources. 


Successful tools and best practices often evolve 
and improve when custom tools are easy to create, 
modify, and share. Although web applications are read- 
ily accessible, only advanced users possess the skills to 
create them. Our goal with A1 is to provide an environ- 
ment where most system administrators can create, 
share, and reuse custom web-deployable tools that inte- 
grate information from different systems. We want to 
leverage existing, common scripting abilities so that 
more administrators can create interactive web apps 
“for free,” without having to become experts in cgi 
scripting or J2EE servlet programming. We also believe 
that shareable, web-based tools will be of significant 
value in aiding system administrators to collaborate. 


Design 


The primary challenges in creating a scripting 
language and environment for system administration 
are ease-of-learning to lower the barrier to entry for 
busy system administrators, providing broad solutions 
in heterogeneous environments, and enabling script 
sharing and reuse for teams of administrators. 


To address these challenges, we took a spread- 
sheet-based approach. Over several decades, spread- 
sheets have proven to be easy-to-learn tools, particu- 
larly because of their straightforward programming 
style. Many system administrators already use spread- 
sheets as part of their regular work. The spreadsheet 
programming model encourages incremental coding, 
putting part of a calculation in one cell, using the out- 
put in a formula in another cell, and so on. Spread- 
sheets are also more robust than most scripting lan- 
guages — the nature of spreadsheet execution means 
that errors only affect dependent cells, and do not nec- 
essarily invalidate the entire sheet. 


The grid simplifies visual layout significantly, a 
common problem for less experienced programmers. 
Finally, the co-location of values and formulas in cells 
enables users to select a value and see the formula that 
defined it. This can help users understand and modify 
others people’s sheets for their own use. 


Al extends traditional spreadsheets by (1) per- 
mitting cells to contain arbitrary Java objects, in addi- 
tion to numbers, strings, etc., (2) extending cell formu- 
las to include calls to methods of cell objects, and (3) 
allowing cells to contain procedural code blocks 
whose execution is triggered by events in the sheet. As 
with most changes that increase functionality, these 
extensions impact ease-of-use, but we believe the 
impact is small and the benefit large. Our use of Java 
takes advantage of a large base of existing libraries, 
particularly for IT management. 
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However, it is important to note that in Al users 
do not need a deep understanding of Java. It is suffi- 
cient to know that objects have methods that can be 
called to change or return information about the state 
of the object. Procedural code blocks are triggered in a 
conceptually straightforward manner, either by 
changes to the value of specified cells or by cell for- 
mulas that evaluate to true. The conceptual model for 
execution is straightforward: “when something hap- 
pens, perform these actions.” The procedural code 
itself is written in a simple scripting language, with 
constructs for assignment, branching, looping, and cell 
method calls. Users familiar with shell scripting would 
find the Al code language similar, and easy-to-learn. 
We also included traditional spreadsheet help features, 
such as pop-up menus on cell objects, so that users are 
not required to memorize all commands, functions, or 
methods — greatly improving Al’s  learn-ability 
through exploration. 

Al can be used to create broad solutions, tying 
together multiple scripts across different languages 
and systems. Within an Al spreadsheet, each cell can 
invoke a different script and display its output, inte- 
grating the output of multiple tools or systems in one 
spreadsheet. Spreadsheets also bridge the gap between 
command-line scripts and graphical interfaces. With 
Al, users can push script output into charts and 
graphs, easily creating data visualizations of critical 
system status and performance. Data visualizations are 
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currently under-used simply because they are too hard 
to create in most command-line environments, and 
vendor-supplied visualization tools are seldom suffi- 
ciently customizable. 


However, Al brings together useful qualities of 
both command-line (efficiency) and graphical interac- 
tion (ease-of-use). The spreadsheet layout also pro- 
vides ample room for comments and explanation that 
can be co-located with command output. While all 
scripting languages permit diligent users to add their 
comments to help others understand script execution, 
such comments are often sparse due to programming 
under time pressure and the fact that comments do not 
improve script execution or usability; the beneficiary 
of comments is usually some one other than the script 
author. Adding comments to a spreadsheet immedi- 
ately improves the sheet’s usability because the com- 
ments are visible at execution time and can help clar- 
ify the sheet’s operation. 


To better enable script sharing and reuse, the Al 
spreadsheet execution engine supports multiple ren- 
dering platforms. Spreadsheets are created using a 
stand-alone client-based tool with full interactivity, 
however the same spreadsheets can be saved to a cen- 
tral repository and executed without modification as 
J2EE-based portlets on a web server. We have inte- 
grated Al with the ISC portal server, which IBM 
developed as a one-stop portal for system administra- 
tion. ISC includes many vendor-provided portlets for 
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Figure 1: A JVM memory utilization monitor running in the Al Client. 
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managing middleware products. Spreadsheets created 
by Al will appear much the same as vendor-supplied 
system management portlets, with the additional 
option to edit existing tools or create new tools on-the- 
fly, extending the toolset with no installation and 
deployment requirements. 


The Al User Interface 


As shown in Figure 1, the Al interface appears 
very much like other spreadsheets, with a grid of cells, 
a cell expression editing field, toolbars and menus. 
Indeed, Al supports all standard spreadsheet function- 
ality: cells can contain strings, numbers, dates, and 
formulas that define a cell’s value as a function of 
other cells. The primary difference is that Al also sup- 
ports Java objects as first-class cell contents, object- 
methods in cell expressions, and event-driven proce- 
dural code in cells. Al’s enhancements to the spread- 
sheet language and execution engines enable users to 
easily build powerful tools that interact with live sys- 
tems. Al spreadsheets may also be deployed as web- 
based portlets. These features are described detail in 
the following sections. 


Objects As First-Class Cell Contents 


To support system administrators’ needs for con- 
trol of external systems, Al adopted an object-based 
approach. Cells in Al can contain any Java object, and 
cell formulas can refer to object methods. For exam- 
ple, if the cell A1 were defined as follows (in this 
paper, cell definitions will be shown with the cell 
name in brackets, followed by the contents of the cell): 


[Al] java.util.LinkedList () 


then A1 would hold a Java LinkedList object from the 
java.util package. Other cells can refer to this object’s 
methods, for example A2 might be defined with a for- 
mula such as: 


{A2] = Al.size() 


so that A2 would always display a value indicating 
A1’s linked list’s size. References to objects and cell 
values can be mixed, for example cell A3 might be 
defined to use the LinkedList get() method, which 
returns the element at the specified index: 


[A3] = Al.get(A2-1) 


This would cause cell A3 to hold whatever value is at 
the end of the linked list (the list is zero-based, so the 
(size-1)"" element is the last). Note that Al uses weak 
typing when matching objects to parameters. For 
example, numbers are automatically converted to 
strings and vice versa depending on the context and 
methods involved. From a programming standpoint, 
enabling cells to contain Java objects gives the user 
access to a large library of objects that provide support 
for programming needs, such as containers, I/O, statis- 
tics, data processing, and networking. From a system 
administration standpoint, users can immediately lever- 
age a variety of existing administration APIs imple- 
mented in Java. 
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One important issue is how to render Java objects 
in a spreadsheet. In Al, each object class is associated 
with an interactor that specifies how it will be ren- 
dered on the screen and how the user may interact with 
it. Default interactors have been defined for most com- 
mon object classes, though A1 allows users to change 
these defaults. Users can also define interactors for 
new object classes in their plug-in objects. Al sup- 
ports graphical rendering and interaction with objects. 
For example, a Boolean object can be rendered graphi- 
cally as a CheckBox, or a NumberCollection can be 
rendered as an X-Y plot. If there are no interactors 
defined for an object, it is rendered textually using the 
default toString() method available in every Java class. 


Interactors are designed to work on multiple plat- 
forms. In the current implementation, Al supports 
three platforms: Java/Swing-based client, HTML-based 
portal server, and text-based command-line. Each inter- 
actor defines how to render and process events in each 
platform; for example, on the Java/Swing platform the 
interactor renders a Button and handles button presses 
in a standard GUI panel, and on the portal server plat- 
form the interactor automatically renders the Button as 
an HTML form input element. 


For improved ease-of-use for those unfamiliar 
with Java, Al provides a library of Java objects cus- 
tomized for spreadsheet use with a simplified set of 
methods and predefined interactors. These objects can 
represent rich data types (e.g., collections, queues, 
stacks), connections to external systems (e.g., SSH, 
SNMP, JMX), graphical widgets (e.g., Button, 
TextBox, ComboBox), and visualizations (e.g., X-Y 
plot, pie chart). Toolbar buttons exist for creating these 
objects, allowing the user, for example, to point and 
click to create an object that manages a server connec- 
tion. 


Al includes a plug-in API for incorporating new 
Java objects into the framework. Experienced Java 
programmers can use this API to enhance existing 
objects with new interactors or with more sophisti- 
cated triggering mechanisms (e.g., pushing events into 
the spreadsheet, externally triggering spreadsheet re- 
evaluation). The API also allows customized objects 
to be added to the toolbar, permitting easier use of 
new objects in the spreadsheet. We believe the success 
of Al will depend to a large degree on a rich set of 
domain-specific objects permitting access a wide vari- 
ety of systems. We have provided a core set of spread- 
sheet friendly objects, but we hope that users take 
advantage of the API to create and share many more. 


Event-driven Procedural and Functional Code 


We found the strictly functional programming 
model of traditional spreadsheets insufficient in defining 
sophisticated control flows necessary for developing sys- 
tem administration tools. To enable richer control flow, 
Al extends the spreadsheet language to permit cells to 
include event-driven procedural code blocks. These 
“micro-scripts” usually include a trigger indicating when 
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the procedural code block should be executed, either by a 
change in value for any of a list of cells, e.g., 


on (<cell address list>) 
{ <procedural code block> } 


or a boolean expression written in the same form as 
cell formulas, e.g., 


when (<boolean expression>) 
{ <procedural code block> } 


Continuing the example above, we might add code to 
clear the LinkedList contents automatically when its size 
exceeds 10 elements by putting the following in cell A4: 


[A4] when (Al.size() > 10){Al.clear()} 


When the object in cell A1 changes, the size method is 
called. If the size is greater than 10, the linked list is 
emptied by the clear method. In general, code blocks 
can be triggered to execute upon changes to cell val- 
ues, clock ticks, button presses, or any spreadsheet 
events. Procedural code blocks are written in a simple 
scripting language containing one or more semicolon 
delimited statements. A statement can call an object 
method (e.g., A1.clear()) or assign a new value to a cell 
(e.g., A10 = A3 * 10). 


There is also support for conditional branching 
using an if statement, and iteration with a for statement 
(both of which work as in C or Java). When code 
blocks have no trigger expression, they must be 
explicitly executed from other code blocks by using 
the cell address as a statement (e.g., A4()). Code blocks 
may also force formula re-evaluation and code execu- 
tion by using the touch statement (e.g., touch A1), 
which triggers all dependent cells as if the cell value 
(e.g., A1) has changed. 


It is important to note that in Al users can assign 
names to cells to improve readability of the code. For 
example, code in A5 can be named “insert”: 


[A5] insert: {Al.add("10.0") } 
and referenced in subsequent code blocks: 
[A6] when ( Al.size() < 5){ insert() } 


For very small tools, using cell references is sufficiently 
clear. However, as tools grow in complexity, naming 
cells considerably aids readability and sharability. 


Through these event mechanisms, users can 
achieve rich control flow in their programs. A spread- 
sheet cell can contain a button that, when pressed, trig- 
gers code to connect to a server and perform some 
action. Regularly polling a remote system can be 
implemented using a timer object that fires at speci- 
fied intervals, with polling code triggered by the timer. 
Any object that uses the Al plug-in API can also push 
external events into the spreadsheet, triggering reeval- 
uation of formulas and execution of dependent code. 


In addition to for loops, iteration is also supported 
in Al through aggregate calls and recursive condi- 
tional constructs. For example, to reset a number of 
servers in cells C1 through C5, one would use the cell 
range notation (“..”) in the method call, e.g., 
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**(A1..C5).reset()”". For conditionally bounded iteration, 
Al’s event-based approach provides a simple solution 
through self triggering when() constructs, e.g.: 


when (Bl < 10) {Bl = Bl + 1} 


Web Portal-Based Collaboration Support 


To support collaboration among system adminis- 
trators, Al spreadsheets can be deployed as portlets in 
a J2EE-based web portal server. In the web portal, 
administrators can execute or customize tools 
deployed by their colleagues from their web browsers. 
The web portal approach greatly helps shared situa- 
tional awareness, as different sysadmins can see the 
same view of system status and controls. 


All Al spreadsheets are initially created in the 
Java Swing-based client application, which can be 
launched from within the web portal framework using 
Java WebStart. For example, Figure 2 shows the tool 
from Figure 1 running as a portlet. From the client 
tool, users can either save the spreadsheet locally (for 
their own use or further development), or to a server 
repository (for web deployment and sharing with oth- 
ers). Once in the server repository, spreadsheets may 
be run in a browser, like any other portlet. When a 
user wants to change a spreadsheet, they can launch 
the Java Swing-based client tool again using the 
“Edit” button on the portlet. When the sheet is 
updated, the user may either save the sheet locally, 
replace the existing portlet, or save it to the portal 
under another name. When a sheet is deployed to the 
portal and running in a web browser, objects are ren- 
dered and manipulated using the HTML form-based 
interactors. In this environment, spreadsheets are ren- 
dered to look no different than any other portlet appli- 
cation. To accomplish this, portlet cells that contain 
code are hidden from the user. Users can also explic- 
itly hide other cells as appropriate. 


Implementation Challenges 


Implementing Al involved several challenges, 
including building the underlying execution engine, 
and developing rendering and interaction techniques for 
both Java application and a J2EE-based environments. 


The execution engine for a traditional spread- 
sheet relies on a dependency graph, recomputing each 
value when there is a change to dependent cells. Side 
effects are not permitted, and there are no guarantees 
as to the number of times that a formula might re-exe- 
cute when triggered. Since Al models real-world sys- 
tems, we must handle side effects properly and ensure 
that code runs only when necessary. In addition, for- 
mulas might include method calls to remote systems 
that might not return results for an arbitrarily long 
time. To support these requirements, we implemented 
a custom multi-threaded execution engine. 


One of the biggest challenges was the need to sup- 
port vastly different interaction models such as the 
HTML form-based interaction and event-based Java/ 
Swing GUI interaction using a single language and 
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framework. In Swing applications, users can interact 
with GUI widgets that immediately generate events and 
cause objects to be rendered to reflect changes. In the 
HTML form model, input elements are contained in 
forms that are submitted and processed by a web server. 


The difference is that in the web model users can 
make multiple changes on the form (such as entering 
text into multiple text fields) and submit the all the 
changes at once using a button on the form. This mini- 
mizes server round trips. Al handles these two differ- 
ent models uniformly through the use of platform 
independent interactors and through “batched event 
propagation.’ Batched event propagation registers the 
current values of all “delayed” input elements (such 
as text input elements) on a portlet. Upon interaction 
with an element that causes a server request, all 
delayed input elements are processed first, before pro- 
cessing the element that initiated the request. Through 
this approach, Al accommodates HTML form-based 
interaction much like the Swing GUI event model. 


Sample A1 Tools 


Simple System Monitor with SSH 


This example demonstrates an Al SSH object 
connecting to and executing commands on a remote 
server (Figure 3). The sample tool includes cells that 
contain labels and text fields for server name, login 
name, as well as a password field object used to avoid 
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storing passwords in the script, and to hide them as 
users are entering passwords: 

[Bl] "j2.almaden.ibm.com" 

{[B2] "eben" 

[B3] PasswordField() 


The SSH object is created in cell B5, referring to 
the values from the above cells: 


[B5] SSH(B1,B2,B3.getText()) 


Once the connection is successfully established, it can 
be used to execute commands and assign command 
output to a spreadsheet cell. Consider the example of a 
system where occasionally a problem causes the 
HTTP server processes increase in number. The sysad- 
min may want to occasionally check to see how many 
of these processes are running using the ps command. 
A button labeled “run” is placed in cell A6, with code 
in B6 that uses the SSH execute() method to run the 
command whenever the button is pushed, putting the 
results in cell C6 (Figure 4): 
{[A6] Button("run") 
[B6] on (A6) {C6 = B5.execute 

("ps -ef | grep http") } 
The example above can be extended to automatically 
restart the HTTP server and send email notifications. 
First, we replace the button with a clock object set to 
fire every 10 minutes, triggering cell B6 which updates 
cell C6 with the number of http processes. This, in 
turn, triggers B7, which checks if this number is 
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Figure 2: The JVM monitor running as a web portlet. 
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greater than 100. If so, it executes a command to 
restart the http daemon, putting any output in cell C7, 
and notifies the user by email by manually triggering 
the code in the cell named “notify” (a.k.a. B8). 


£S$ESBxaA a uP CIM =] oI FE 
Bs sds = SSH(BL, B2, B3.getTexti)) 


= 


j2.almaden.ibm.com 


eben 


| 

| 
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| | 3 password | ee — - 
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Figure 3: Creating an SSH connection. 


[A6] Clock(10*60*1000) 
[B7] when (C6 > 100) {C7= B5.execute 
("/etc/init.d/httpd restart"); 
notify()} 
[A8] MailService ("j2.almaden.ibm.com") 
([B8] notify: {A8.sendMail("root", 


"“eben","http server", 

"server restarted"+tc7) } 
This example shows how Al permits the user to 
quickly create a web-deployable tool that displays and 
controls the status of a remote system. This tool can be 
easily shared with other system administrators, who 
need only change the server, login, and password values 
to run the script within their own server environment. 


Graphical JVM Memory Monitor 

Figures | and 2 show a tool that uses Java Man- 
agement Extensions (JMX) to connect to several Web- 
Sphere Application Servers (WAS) and display their 
Java Virtual Machine (JVM) memory usage using a 


n 
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pie chart. In this example, we show how to create that 
tool in detail. 


First, we create labels in column A, and 
TextField objects in column B into which the user can 
enter the WebSphere server name and JMX listener 
port. The object name is visible when the user is typ- 
ing. Once complete, the object is rendered, in this case 
as a fully interactive text field (see Figure 6). 


Next, in cell B3 we create an object that encap- 
sulates a JMX connection to the WebSphere server: 


[B3] WAS(B1,B2,"","") 


The code creating the WAS object refers to the values 
that the user types in to cells B1 and B2, and uses 
empty values for login and password (as authentication 
for JMX is turned off). In JMX, servers can return one 
or more MBeans, which contain a number of attributes 
and methods. To retrieve an MBean from a WAS JMX 
object, we will call the getResource() method in B4, 
specifically asking for the JVM MBean: 


[B4] = B3.getResource ("JVM") 


If users can’t remember all methods available on an 
object, it is possible to right-click on the object to pop- 
up a method list (Figure 6). Once we have the JVM 
MBean, we can get further information by using meth- 
ods such as getTotalMemory() and getFreeMemory() 
to get the memory statistics into cells B5 and B6, 
respectively: 

[B5] = B4.getTotalMemory () 

[B6] = B4.getFreeMemory () 
It turns out that the JVM MBean doesn’t have a 
method for returning used memory, but that is easily 
computed from the difference between total and free 
using a regular spreadsheet expression in cell B7: 

[B7] = B5 - B6 
Now we have cells that show total, free, and used 
memory. Because this particular API does not support 
“pushing” data from the server, we need to regularly 
poll the server to keep the values up to date. This is 
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Figure 4: Running an SSH command triggered with a button. 
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Figure 5: Automating HTTP server restart. 
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accomplished with a timer and code to trigger updates. 
Cell A9 holds the Clock object, 


[A9] Clock() 


and cell B8 contains code that, whenever the clock 
fires, “touches” cell B4, causing all dependent cells to 
be reevaluated: 


[B8] on (SAS9) { touch(B4) } 


Note here that we used the “S$” modifier in the cell 
reference to make it an absolute cell address, thus 
ensuring that when cell is copied to another location, 
the expression will still refer to cell A9. This will make 
it easier to duplicate the code for use with other 
servers. Now the memory values are being updated at 
regular intervals. The display can be improved, how- 
ever, since textual number displays aren’t the best way 
for displaying relative values such as used and free 
memory. We now create a pie chart object in cell B11: 


[Bl1] PieChart() 


To populate the pie chart with values, we add code in 
cell B9 to have a pie slice with a name from A6 
(‘‘Free’’) and value from B6: 


[B9] on (B6) { B$1ll.set(S$A6, B6) } 


Thus, whenever the value in B6 changes, the pie slice 
is updated. In the reference to B$11, only one ‘*S”’ is 
used in the cell reference, making the row of the chart 
invariant. This permits us copy/paste cell B9 to B10, 
where it will create a pie slice for “Used” memory, 
and then copy both cells to other columns for other 
servers. The reference to $A6 makes the column 
invariant, since the labels are only found in the first 
column. We now have a complete memory monitor for 
a single server. Expanding it to support more than one 
machine is relatively easy. We just select cells B1 
through B11, copy and paste them into columns C and 
D. Al supports standard spreadsheet cell expression 
rewriting, so non-absolute cell references in formulas 
are changed when pasted to refer to the new columns. 


The working memory monitor is shown in Figure 
1. The memory monitor can also be saved to a server 
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repository where it becomes immediately available as 
a portlet tool. Figure 2 shows a snapshot of the portlet 
tool as it appears running in a web browser. 


Creating a Server Inventory Report 


We have observed system administrators receiv- 
ing management requests for inventory reports of all 
servers owned by an organization. The requests came 
from different people, and involved different groups of 
servers. The requested information included OS type, 
version, platform, hardware details (processor 
type/speed, memory), storage utilization, network 
interfaces, machine location, etc. The sites involved 
had various versions of Windows, UNIX, Linux, and 
OS/2, so capturing the required information required a 
variety of mechanisms, e.g., SSH, SNMP, manage- 
ment tools, and databases. The system administrators 
we observed collected such information manually, 
entered into a spreadsheet, created necessary charts, 
and sent the report to management. Given the nature 
of this task, it seems ideal to use Al to both retrieve 
and organize the data. 


Here, we show two examples of such data collec- 
tion, one in Windows through SNMP and the other in 
Linux through SSH. Each machine is listed on a sepa- 
rate line, so system administrators can easily duplicate 
it for as many servers as necessary through copy and 
paste. From these examples, it should be clear how 
other server and OS types and access methods could 
be implemented. 


First, we create a set of buttons to trigger updates 

on system information (Figure 7). In cell A1, we put a 
button to update all server information, and each line 
corresponding to a server also has an individual button 
to specifically update that server’s information. 

[Al] Button("Update All") 

[B4,B5,...] Button("Update") 
Cells C4, C5, and so on contain code to trigger that 
row’s button whenever the master button in A] is hit. 
For example, C4 will contain: 
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[C4] on ($A$1) {touch(B4) } 


For each machine, column A will be the machine 
type/access method for the convenience of the spread- 
sheet user copying and pasting rows (e.g., Win- 
dows/SNMP, Linux/SSH), and Column D_ has 
approach specific code to connect to remote server. 
For SNMP and SSH the code is: 


[D4] on (B4) { E4 = SNMP(I4, "", um) } 
[D5] on (B5) 
{ E5 = SSH(I5, G5, H5.getText())} 


Whenever the button is pressed, a connection object is 
created, causing other values in the row to be updated. 
Connection status is reported in the fifth column with 
a simple method call: 

[F4] = 
The subsequent three columns include login/password 
(if needed), and host name. The following columns 
contain formulas that derive machine information from 
the connection object. With SNMP, we get the machine 
description, uptime, #network interfaces, and network 
interface descriptions as shown below, respectively: 


E4.testConnection() 


E4.getResourceValue 
(8123: 62172 sletal 
E4.getResourceValue 
CU 3's 655. 12 a Tiegh 
E4.getResourceValue 
CUlaS: by Le? cated 
E4.getResourceValues 
CYL Be Bila 2's hed 2Ble 2°") 


The same information can be retrieved from a Linux 
machine via the following use of the SSH object: 


sii") 


ll 


«3% 0") 


ll 


- Oi) 


= E5.execute("uname -osrv") 
E5.execute("uptime") 
E5.execute("netstat -i | we -1") - 2 
= E5.execute("netstat -i") 
Other pieces of system information may be extracted in 
a similar fashion, though some require more extensive 
textual processing, either using shell commands 
through SSH, or using one of the built-in string process- 
ing functions provided by Al. Though at first it may 
seem to be complex, once a row collecting information 
for one machine is completed, it one can be copied and 
pasted for all other machines of the same type. 


Evaluation 


Al is currently an alpha release. We completed 
one laboratory usability study and are currently con- 
ducting field trials. Our laboratory study involved 
twelve participants, seven professional sysadmins and 
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five programmers. Of the seven system administrators, 
two had more than four years scripting experience, and 
the remaining had zero to two years experience. One 
goal was to determine whether Al could be learned 
and successfully used in a two-to-three hour period. 


In the study, participants first reviewed a self- 
paced tutorial of the Al programming language and 
user interface. Second, participants practiced using Al 
by developing a simple tool following explicit step- 
by-step instructions. The first two steps typically took 
about 30 minutes each. Next, participants were asked 
to develop two system administration tools, one based 
on the other, given only descriptions of the tool 
requirements. Finally, an interview was conducted 
with participants to learn about their experiences. 


The first tool to be created was a log-space-mon- 
itoring and backup-automation script for an http 
server. The tasks included querying the http server for 
the name of its current log disk, using that name to 
query the file system for log disk utilization, and reg- 
istering listeners to monitor disk utilization. When the 
log disk is nearly full, code must notify the system 
administrator through email, and when the log fills 
completely, it must stop the http server and start a disk 
backup. The second tool extended the first one 
through code to switch log disks automatically when 
two log disks are available. 


After taking the tutorial and building the sample 
tool, the participants spent between one and two hours 
working on the subsequent tools. Overall, they were 
fairly successful, on average completing 80% of the 
required functionality. Given the complexity of the 
tools and the limited time to learn the language and 
environment, these results are very encouraging. 
Equally encouraging were positive comments by the 
participants on the incremental and interpreted nature 
of Al. One said, “It allows me to do things in my own 
order. I can refer to a non-existing object. I know I’m 
going to create the object next.”” Another said, ‘The 
fact that I can interact with my systems in real time in 
the spreadsheet that alone is pretty cool.” Another 
added that building tools this way was “the quickest 
live application [I] ever made.” 


The results did highlight several areas for 
improvement. Learning the object-oriented program- 
ming model proved somewhat of a hurdle for the less 
experienced participants. In addition, many partici- 
pants had difficulty understanding the semantics of the 
objects and methods used in the tools. These problems 
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Figure 7: An inventory spreadsheet. 
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suggest a need for improved introductory documenta- 
tion to explain the programming model, and better 
interactive browsing of object APIs to aid program- 
mers in learning object and method semantics. 


We are also conducting field trials with system 
administrators, both inside and outside our company, 
to see if they can develop tools that are useful in their 
work settings. So far, Al has been field-tested in three 
different groups: web-hosting, server management, 
and cluster management. The study primarily included 
requirements gathering, use-case and template devel- 
opment, first-hand experience by system administra- 
tors in tool building, and data collection regarding 
problems and issues with Al user interface, language, 
environment, and libraries. In these, Al was used for 
developing tools for recycling a steady-state server, 
creating system inventory reports, and monitoring 
cluster server performance. 


Overall, the reaction to Al was quite positive. 
Users especially liked live connections to systems, 
easy integration of data, and the web interface. One of 
the users said: “*... but this is like you have real time 
technical data, and accurate.” Integration was a major 
concern among users in their daily work. One user 
said: “*... this is the only tool that brings all this data 
together.” Another referred to the difficulty of creating 
web interfaces in one of their projects: “[to get a] web 
front in front of it (the report) it took six months and it 
still didn’t work right,” adding “[with Al] it was on 
the fly, it is great!”” Users also appreciated the fact that 
Al does not require installing agents on servers, 
pulling data from systems. Issues concerning Al 
including lack of support for running scripts as back- 
ground jobs, need for more interactive scripting and 
improved text processing utilities, and lack of suffi- 
cient documentation and online help on object libraries. 


Related Work 


Scripting is pervasive in system administration, 
primarily through command-line shells. System 
administrators often open a number of shell consoles, 
execute command-line instructions and automate tasks 
through languages such as Perl, Python, and C Shell 
[10]. System management tools are typically vendor- 
specific-each vendor provides a management tool for 
its own systems, such as IBM’s DB2 Control Center. 
Tivoli and EMC, two leading providers of tools for 
system administrators, have hundreds of tools for 
enterprise-wide operations. To address the fact that 
system management activities often span multiple 
tools from multiple vendors, both Tivoli and EMC 
have integrated environments. For instance, Tivoli’s 
Enterprise Console provides comprehensive manage- 
ment capabilities, including monitoring systems from 
multiple vendors. 


One limitation of these environments is that they 
provide little support for customization. For example, 
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the systems typically offer standard, prepackaged 
charting capabilities without a means for end users to 
create their own system views. PIKT [13] and 
Cfengine [3] are well known in the system administra- 
tion community as domain-specific languages for 
monitoring and configuration management of diverse 
environments. Unfortunately, complexity of these lan- 
guages and lack of supporting visual environments 
make them suitable mainly for experts. 


Related work on spreadsheets is extensive in 
both the research and commercial realms. Spread- 
sheets were one of the original “killer apps” that 
prompted wide use of personal computers. While 
effective for tabular calculations, traditional spread- 
sheets lack expressibility and programming power [17, 
4] and they are limited in their ability to interact with 
external systems. Researchers have taken a variety of 
approaches to these problems, investigating changes to 
spreadsheet language, programming model, data 
types, and user interface. 


Of particular relevance is work on object-ori- 
ented spreadsheets [2, 5, 9]. These approaches use 
object models specific to their systems, and require 
new languages for object definition. Our use of Java 
has the advantage of a large base of existing users, 
tools, and libraries, particularly for IT management. In 
addition, the other approaches were purely functional 
programming environments, whereas we think our 
procedural code blocks provide a more natural means 
of managing the side effects (such as restarting a sys- 
tem) inherent in system administration. 


Commercial spreadsheet systems (e.g., Microsoft 
Excel) address limitations in programming power and 
access to external systems by permitting users to write 
programs in an external language (e.g., in Visual 
Basic). This approach has several drawbacks. First, 
data sources and processes are not explicitly repre- 
sented in the spreadsheets as first-class cell contents 
(unlike numbers, strings, etc.). Thus, interaction with 
external processes is done through a language and pro- 
gramming model (e.g., Visual Basic) different from 
that used in the spreadsheet and typically requires dif- 
ferent programming skills. Second, functionality pro- 
vided by external processes (e.g., operations defined 
on servers, such as shutdown) may not be exposed to 
the user in a form that can be used readily in spread- 
sheet expressions. Finally, data are typically pulled 
from external processes rather than actively pushed by 
these processes (though in Visual Basic it is possible 
to program this explicitly too). These issues are all 
important for system administration. 


Al extends previous work through explicit repre- 
sentation of external systems as objects in cells. These 
objects expose their properties and methods to the user, 
who can use them in functional expressions or procedural 
code blocks to query and control systems. Unlike previ- 
ous approaches, Al carefully combines procedural and 
functional constructs through an event-based approach to 
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achieve rich control structures for the system administra- 
tion domain. Al takes the inherently event-based pro- 
gramming of spreadsheets to the extreme, where each 
cell can either explicitly or implicitly create, listen, com- 
bine, and process events. Most importantly, Al’s exten- 
sions are carefully crafted to create a model and language 
that conforms to the spreadsheet metaphor while remain- 
ing usable, and powerful. 


Discussion and Future Work 


Al is a working prototype that aims to support 
system administrator tool building. There remain open 
questions as to how well it will work in the real world. 
We are currently involved in field trials to answer spe- 
cific questions: 

e Is the Al environment sufficiently easy to use 
(and reuse) to permit a broader population of 
system administrators to do their own script- 
ing? 

e Is the Al environment sufficiently powerful 
and usable to be useful for real system adminis- 
tration tasks? 

¢ Can Al be learned quickly enough so that 
administrators are willing to make the time 
investment to learn it? 

¢ How valuable is a web portal repository of 
shared tools? 

¢ Does Al have any scaling limitations for real 
world tasks? 

e What set of access methods (SSH, SNMP, 
JMX, etc.) is needed? 


We hope to better understand how to make Al 
successful in the real world through our field trials, 
and through a public release of Al. There are several 
areas in which we are already working to improve Al. 
Error handling and debugging of scripts need 
improvement, since our extensions to the program- 
ming model can make dependencies harder to under- 
stand. In addition, right now Al has no mechanism for 
logging spreadsheet events, a necessary feature when 
using Al to manage systems. We are also actively 
seeking to develop more plug-ins for connecting to a 
wider variety of systems through additional protocols. 


Conclusion 


We have developed a tool-building environment to 
support system administrators in working collabora- 
tively to create custom solutions for their site. It pro- 
vides access to a wider audience by combining a spread- 
sheet-style development environment with web portal 
deployment. An early version has been made available 
on the web, and we are actively testing and developing 
it to make Al a useful tool for system administrators. 


Availability 


Al is scheduled to be available on IBM alpha- 
Works in October 2005. 
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HostDB: The Best Damn 
host2DNS/DHCP Script Ever 
Written 


Thomas Limoncelli — Cibernet Corp. 


ABSTRACT 


HostDB is a system for generating DNS zone files, BIND configurations, and ISC DHCP 
server configurations. It is extremely simple yet powerful, which makes it easy to deploy. It is not 
so complicated that it requires an SQL database or web server. It is not a single generation script 
but a complete system that includes tools for deploying the files that are generated. The system is 
written in Perl and shell and makes heavy use of make. The zone files that it generates look hand- 
made, except that they are clean. Powerful tools are also provided to aid in the conversion from 
legacy systems. HostDB is the third generation of my DNS maintenance systems and embodies 
not just superior software but also best practices in system administration. HostDB uses small 
tools that combine to achieve large goals. Initial deployment can be done in stages to reduce risk. 


Deployment of updates is fully automated. It rocks. 


Introduction 


The goal of HostDB is to create a DNS zone 
generator that is full-featured without requiring com- 
plicated databases and web services. The system uses 
simple configuration files and automates deployment. 
It is easy to deploy whether you are converting a 
legacy system one zone at a time or setting up a new 
system from scratch. 


Many small and medium sites (dozens or hun- 
dreds of machines) hand-edit DNS zone files. This is 
an error-prone practice which requires a high skill 
level. The smallest typo can cause big outages. 

Many large sites (multiple IT teams, locations, 
and internet connections) use DNS management sys- 
tems that provide no revision control and very little 
audit trail of who made what changes. Many also 
require external (outside the firewall, or ‘‘public”’) 
zone files to be maintained by hand, which adds the 
risk that they will become out of date with internal 
(inside the firewall, or ““private’’) zones. 


The problem with most DNS generation scripts 
is that they are either too simplistic to be useful or so 
complicated that they are intimidating to install. The 
simple ones are fine for doing an initial conversion 
from /etc/hosts to zone files that will be hand-edited in 


the future. Many are “point tools”’ that require a lot of 


integration work. The complex ones are so unwieldy 
that small sites find them too scary to install. Most 
don’t handle complicated DNS idioms such as having 
special MX records for mail servers, proper handling 
of multi-homed hosts, or templates for ranges of simi- 
larly named hosts. They don’t handle generating dif- 
ferent zone files for inside the firewall versus outside 
the firewall (i.e., “host hiding’) resulting in hand- 
edited external zone files that are prone to errors. 


Many generate zone files that are not pretty: hand- 
edited zone files would be more readable (if we only 
had the time to maintain them that way!). HostDB 
fixes all these issues. It does this without introducing a 
complicated syntax plus it generates your DHCP con- 
figuration file “for free”! The code is written in Perl, 
shell, and make and is freely available. 


Setup time is fast. Really fast. The system can be 
used to generate zone files that can be individually 
inserted into your legacy systems within minutes of 
installing the software. HostDB breaks from the Free 
and Open Source Software (F/OSS) tradition of pro- 
viding a default configuration that is an arrogant dis- 
play of the most complicated features. Instead the 
default configuration is a useful example that can 
often by used “out of the box” with only small 
changes. Documentation is included for migrating 
from a legacy system in small steps with plenty of 
testing along the way. 


Audience 


HostDB seeks to fill the void between simple 
DNS zone generation scripts and massive DNS/DHCP 
database-driven applications. It is perfect for a site that 
is starting fresh, or for one that currently hand-edits 
zone files and wishes to introduce automation. 


The learning curve and installation curve for 
HostDB is very small. Thus, it is appropriate for sites 
that do not have the need, expertise, or resources 
required by systems that have a web-based front-end 
or use a SQL database as the backend. 


HostDB is good for sysadmins that need an 
extremely good solution but don’t have a lot of time. 
Migration from a legacy system is extremely easy 
because the files that are generated are put into a 
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repository before they are copied to their final destina- 
tion. Thus, comparisons between the legacy zones and 
the HostDB-generated zones can be done as part of 
acceptance testing. Tools for comparing zones are pro- 
vided. Individual zone files can be put into service 
permitting an incremental deployment methodology 
(rather than a “‘flash-cut’’), thus reducing risk. 


HostDB is not appropriate for extremely large 
organizations with many separate IT groups or sites 
with complicated dynamic DNS configurations. Their 
needs would be better served with other, larger, sys- 
tems. It doesn’t directly support a sophisticated autho- 
rization schemes, a web-based interface, or assistance 
for dynamic DNS updates. However, HostDB does 
have some hooks that make it easy for simple dynamic 
DNS to be used. Also, large organizations are often 
composed of many smaller IT organizations which 
would find HostDB entirely appropriate. These orga- 
nizations often have one team that manages all inter- 
net connectivity and HostDB is excellent for maintain- 
ing the related DNS zones. 


At this time HostDB has no support for IPv6. 
I'm okay with this. 


What Does It Do? 


I have seen many exceptional DNS/DHCP con- 
figuration management systems that provide excellent 
web front ends and are so powerful that implementa- 
tion requires an SQL database. HostDB isn’t one of 
those. 


HostDB lets you track all DNS and DHCP infor- 
mation about all your hosts in a single, simple file. 
From that file, it generates DNS zones, BIND configu- 
ration files (named.conf) and host entries that can be 
included in an ISC DHCP server configuration 
(dhepd.conf). Two helper configuration files guide 
zone creation and file deployment, but they are basi- 
cally static after the system is stable. 


The system generates the complete zones and 
configurations. No hand-editing after the fact is 
required. Thus, automation using make and other tools 
is possible. 

The system handles complicated DNS idioms 
such as: 

e Special MX records for mail servers. 

¢ Proper DNS records for multi-homed hosts and 
routers. 

¢ Templates for ranges of similarly named hosts. 

e Host-hiding (generating different zones for 
external DNS servers). 

¢ Clean, human-readable zone files are generated 
(as if they were human-generated, but neater) 


HostDB includes a tool called ““mkdestinations” 
which generates a Makefile that will efficiently deploy 
(push) individual files to local and remote hosts. It 
recognizes zone files and treats them specially to pre- 
vent zone-transfer storms. Non-zone files can be 
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deployed as well, making it possible to maintain the 
all files related to DNS and DHCP configuration for 
multiple servers around a network from one central 
administration host. 


Major tools are written in Perl, with many helper 
programs in BASH shell. Shell glues them together in 
a way that is easy for sysadmins to understand. The 
system tries not to re-invent the wheel, thus it uses 
standard UNIX tools: make, SSH, sed and awk. Sed 
and awk have been feeling ignored lately so we threw 
some work their way. 


How to Configure 


There are three configuration files. Each encap- 
sulates information that would be updated by a partic- 
ular audience. The format is either simple or compli- 
cated based on the audience as summarized in Table 1. 


The syntax of hostdb.txt is simple because a non- 
DNS expert needs to be able to edit it without intro- 
ducing errors. Rather than designing a syntax by and 
for DNS experts, much effort was spent in designing a 
syntax that made sense to the junior engineers that 
would be using the system. The filename ends with 
“txt” to facilitate non-UNIX admins editing these 
files through non-UNIX tools like Notepad. 


zoneconf.txt requires a little more knowledge of 
how DNS zones work, which is appropriate because 
this file should only be edited by a person with 
advanced DNS knowledge. 


destinations.txt is extremely simple because it 
simply states where files are to be copied after they 
are generated. It permits files to be renamed as they 
are copied, which makes inserting a generated file into 
a pre-existing legacy system very easy. 


Standard UNIX protections can be used to con- 
trol who can edit these files. They are all plain ASCII 
files, appropriate for maintaining under revision con- 
trol systems like RCS, SCCS, CVS or Subversion. 
Any system administrator maintaining key files such 
as these without a revision control system is taking an 
undue risk. Revision control means knowing what 
changed when. When a change is introduced that 
causes an outage, one can undo a change, investigate 
who made the change, mentor them so they do not 
repeat the mistake, and optionally ridicule them during 
staff meetings. 


DNS Zone Generation 


DNS zones are generated using “mkzones”. 
Mkzones parses the entire hostdb.txt file, storing the 
important bits into various data structures. It then reads 
zoneconf.txt which is “executed” line by line, setting 
options and generating zones as instructed. Because of 
this design, an option can be set one way before a par- 
ticular zone is generated and then another way before 
the next zone is generated. This is a powerful feature, 
letting one system generate zones with very different 
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needs. For example, if one sub-domain requires differ- 
ent default MX records than all the others, it is very 
simple to achieve that goal. Other DNS generation 
systems assume the same options for all zones. That 
makes them inappropriate for very large sites. 


The power of mkzones’ algorithms comes from 
the fact that the parsing of hostdb.txt is decoupled 
from the generation of the zones. This permits more 
sophisticated zone generation since the generation 
algorithms understands “the big picture,” i.e., all host 
information is available at once. 


Decoupling parsing from generation also had the 
benefit that the input syntax could be changed without 
affecting the generation code. We were able to experi- 
ment with different syntaxes over time without worry- 
ing about how this would break the generation code. 


The syntax for hostdb.txt was developed with 
massive user input. Users were presented with options 
for syntax and their feedback altered the evolution of 
the system. Most DNS zone generation systems tend 
to be designed for the DNS expert. HostDB has the 
benefit of a host list which is in a format that is perfect 
for junior sysadmins and the complicated options are 
kept far away in a different file where only the experts 
can make changes. 


The users quickly lined up behind a syntax that 
looked like /etc/hosts, not a fancy recursive syntax like 
ISC’s BIND and DHCP configuration files. They 
requested a single line per IP address because when it 
is time to allocate an IP address (their most frequent 
task) the procedure they are most comfortable with is 
finding an unused IP address in a long list, sorted by 
IP address. They also didn’t want to be bothered with 
having to keep the file perfectly indented or brackets 
matched. 


What Makes the Generation So Cool? 


To make a zone file look readable took quite a 
long time. After many iterations we finally realized 
that the most readable zone file stripped hostnames 
(removed the domain, when possible), were formatted 
with tabs (not spaces), grouped records related to a 
particular host, and ordered the hosts by IP address. 

Therefore, when generating zone files, we output: 


1. The SOA information 
2. The NS records 
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3. Any records for the current zone name (the 
domain without any host) 

4. All the DNS information related to a particular 
host (with hosts listed in IP address order) 


You may notice that the examples from 
hostdb.txt always specify Fully Qualified Domain 
Names (FQDNs). That is, foo.example.com, not just 
simply “foo”. Usually DNS systems assume a label is 
simply a hostname (sans domain) unless it ends with a 
single period. However, humans tend to forget the 
period. After many iterations of trying to solve this 
problem by being fancy, we decided it was better to 
require the users to always enter FQDNs and output 
warnings if something was not a FQDN. 


The decision to group the records in the zone file 
by hostname (yet stay in numerical IP address order) 
was something of a small breakthrough discovered 
after many trials. To do this all DNS information has 
to be in memory before the first zone is generated. 
This way decisions can be made by checking in-mem- 
ory data structures and flags that were built up during 
parsing. Items from the in-memory data structures are 
deleted as each DNS resource-record is output into the 
zone file, thus eliminating the possibility of outputting 
the same line twice. 


BIND Configuration File Generation 


mknamedconf uses the information in 
zoneconf.txt to generate [SC BIND configuration files 
for DNS masters and slaves inside the firewall, plus 
DNS masters outside the firewall. It does not generate 
configurations for slaves outside the firewall (exter- 
nal) because the author has not yet found the need for 
such a thing. External slaves tend to be owned and 
managed by separate organizations, often ISPs. We do 
not know enough to generate their entire bind configu- 
ration, nor can we assume they use BIND. Configura- 
tion updates are usually done via manual (emailed) 
requests of the owner of the slave server. 


DHCP Generation 


The DHCP generation is very simple. Mkzones is 
called with a flag that tells it to parse the hostdb.txt file 
as usual, but then output a ISC DHCP “host” statement 
for each machine with DHCP-related data. This infor- 
mation can be included in a ISC DHCP “‘dhepd.conf” 
file, which must be manually created. DHCP configura- 
tions are usually too custom to warrant generating them 
automatically, except for the host statements. 


Skills required 


hostdb.txt Anyone on your 


sysadmin team 


Info about all hosts 


Edit text files, maintain a format set 
up for them, understand IP 
addresses 





zoneconf.txt DNS engineers 





destinations.txt 


How the zone files and BIND 
configuration files are generated 








Understanding of DNS zones and 
terminology 


UNIX admins setting | Where zone files and other files | Knows where zone files are stored 
up BIND are copied and has root access on DNS servers 


Table 1: HostDB configuration files. 
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The Syntax 


The systems works in two phases. First, the 
zoneconf.txt and hostdb.txt files are used to generate 
all the files that are generated. Next the new files are 
copied to the appropriate hosts/directories based on a 
map described in destinations.txt. 


You might think of this as: 
Hostdb.txt + zoneconf.txt — generated files 
Generated files + destinations.txt — deployed files. 


hostdb.txt Syntax 


The hostdb.txt file looks like a /etc/hosts file 
augmented with flags and parameters. Display 1 
demonstrates the syntax of hostdb.txt: 


The zone file ‘“‘example.com” that would be gen- 
erated would be a simple “A” record for each host 
plus appropriate NS and MX records as configured in 
zoneconf.txt. The reverse zone (1.1.10.in-addr.arpa) 
would be one PTR file as appropriate for each host. 


Notice that each host is listed by its fully quali- 
fied domain name (FQDN). For example, spoon. 
example.com instead of just “spoon”. During user 
testing, it was discovered that it was too difficult to 
come up with a syntax that permitted “short” host- 
names to be used and still permit multiple domains to 
be described in the same file. Various formats were 
attempted. In the end, it was decided that it was 
cleaner to make every host be listed as a FQDN. 


“kettle.example.com” has a special option on it: 
“MAC=00:b0:d0:a6:cf:f1”. This identifies its Media 
Access Control (MAC) address, or Ethernet, address. 
When a DHCP configuration file is generated, this 
host will have a “static assignment” or “permanent 
lease” for 10.1.1.2. We have found it useful to assign 
static assignments to all known hosts, and use “pools” 
of randomly allocated addresses only for transient 
hosts. This makes log files more accurate since most 
machines will always appear at the same IP address. 

“fork.example.com” has a CNAME of “pitchfork. 
example.com’’. This generates a DNS CNAME record. 

In addition to CNAME=, there is also 
ANAME=. This generates multiple DNS A records 
pointing to the same IP address. HostDB generates the 
reverse lookup to be the first host on the line, which 
seems to be the most common practice. 
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Multiple ANAMEs and CNAMEs can be listed 
by separating them by colons. For example one might 
list “‘ ANAME=pitchfork.example.com:branch.example. 
com:divide.example.com”’. 


ANAMEs and CNAMEs can both appear on the 
same host line. 


The plural ANAMES is the same as ANAME, as 
is CNAMES the same as CNAME. This makes the 
hostdb.txt file slightly more readable. 


Multihomed Hosts 


HostDB understands that multihomed hosts need 
special treatment. For example, a router has many 
interfaces. The best practice is to have one DNS label 
or name that will return all “‘A”’ records; one for each 
interface. However network engineers and others need 
to be able to specify a particular interface when they 
want. HostDB likes it both ways. 


Suppose the .1 address of each network was an 
interface for our router, zathras. We could specify that 
as shown in Display 2. 


The DNS zone information generated would be: 


zathras IN A LO edict Os 4 
IN A 1Ox1. 6.205 1 
IN A 10:.2:.30: 1 
IN A 10)5:16: 40%. 1 
zathras-red IN A 10.1.10.1 
zathras-blue IN A 10:..1.210:. 1 
zathras-green IN A 102153105 L 
zathras-purple IN A 10.1.40.1 


(MX info deleted for brevity, but I assure you they are 
awesome.) 


Someone that wanted to reach any interface 
would use “‘zathras”. A network engineer that needed 
to refer to the zathras interface on the “red” network 
would specify “zathras-red.example.com.” 

The reverse-lookup zone indicates the individual 
interface names (zathras-red, not zathras). 

HostDB also has “ISMULTIHOMED=” which 
is the same thing but for servers. Currently it is exactly 
the same thing as “ISROUTER=”.. The separate direc- 
tives are provided should different functionality be 
required for routers and hosts in future versions. 
Host-hiding 

Host hiding is where, for either real or perceived 
security reasons, most sites do not want to expose the 


ISROUTER=zathras.example.com 
ISROUTER=zathras.example.com 
ISROUTER=zathras.example.com 


10:01 oD ok pot.example.com 
10) 7 52 2 kettle.example.com MAC=00:b0:d0:a6:cf:f1 
10),.1 d s3 spoon.example.com 
10.1.1.4 fork.example.com CNAME=pitchfork.example.com 
16)..1 od 33 spatula.example.com 
Display 1: Syntax of hostdb.txt. 
10.1.10.1 zathras-red.example.com 
LO. 1 820.8 zathras-blue.example.com 
10:51 301 zathras-green.example.com 
10.1.40.1 zathras-purple.example.com 


ISROUTER=zathras.example.com 


Display 2: Specifying .1 addresses as a router interface. 
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names of their hosts to the outside world. Inside their 
firewall, a host may be called patentdatabase.exam- 
ple.com, but outside it should be simply known as 
h64-32-179-55.example.com. Obscuring the host name 
like this prevents external users from being able to 
make educated guesses about good attack targets. It 
prevents internal host names from being exposed out- 
side the network, which might prevent embarrassment 
by preventing ceo-pc.example.com from appearing in 
the Apache log files of www.sexyteens.com. 


For forward DNS lookups one generally sets up 
an external DNS zone file that lists only the hosts that 
the public internet needs access to. This is used on 
external DNS servers and is often managed separately 
from the internal of DNS zones. Any time two separate 
databases are used to store information about the same 
thing you are asking for trouble; they will get out of 
sync. It is my experience that even sites with very full- 
featured commercial DNS management systems use a 
hand-edited zone-file for their external DNS zones. 


Reverse DNS lookups are another matter. Some 
sites simply do not provide any reverse-DNS records. 
As a result, other sites refuse to talk to them because, for 
real or perceived security reasons, other sites are weary 
of accepting connections from sites without proper 
reverse DNS. Other sites simply use statically generated 
reverse labels that encode the IP address to keep them 
unique. For example, at this time Optimum Online’s 
reverse DNS for 67.82.128.6 is ool-43528006.dyn. 
optonline.net (the digits are the hex interpretation of the 
IP address, the ‘‘ool” is Optimum OnLine). 


To that end, HostDB generates a different set of 
zone files for ‘‘internal’” DNS servers (those inside the 
firewall) and “‘external” servers (those accessible to 
the public). HostDB generates both sets of zone files 
from the same hostdb.txt, thus preventing them from 
getting out of sync. 


To make this work, HostDB assumes that a host 
should be hidden from outsiders unless marked otherwise. 
To change the default, add a “scope qualifier” to the end 
of the hostname. The scope qualifiers are @EXTERNAL, 

64:32 179.55 


64.32.179.56 
64/32. 21.79.2577 


database. example.com 


vpn3000.example.com 
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@EXTERNALONLY and @INBOUNDNAT. Display 3 
shows a hostdb.txt that demonstrates all three. 


database.example.com is a normal host. Insiders 
have the usual forward and reverse DNS data. Out- 
siders can not look up the host’s IP address, and the 
reverse DNS information is the anonymous name of 
d64-32-179-55.example.com. Lookups of 
d64-32-179-55.example.com return a proper A record. 


www.example.com is an externally exposed host. 
Therefore, the forward and reverse lookups work as 
one would expect for both internal and external users. 
Since “example.com” is an ANAME for this host, it 
behaves the same way. 


vpn3000 and vpn demonstrate the @EXTER- 
NALONLY scope. This keyword exposes a host exter- 
nally but not internally. It is rarely used. In the above 
example we use it for a VPN (RAS) server which 
external people use to access internal resources. Thus, 
they shouldn’t use it when they are inside the com- 
pany. We wanted to be able to construct a client con- 
figuration that would fail when used inside the fire- 
wall, where use of the VPN should be unneeded. Thus 
we marked vpn.example.ccom as @EXTERNAL- 
ONLY so that the DNS entry would appear in external 
zone files, but not in internal zone files. This achieved 
the goal. However, network administrators still needed 
to access the VPN server for administrative reasons. 
They could use the name vpn3000.example.com 
which resolves properly internally. 


The last two rows of the table demonstrate that 
reverse lookups for the anonymous hostnames are 
properly generated. Any A record that is generated 
needs a proper PTR record, even when host hiding. 


Not pictured in Display 3 is @INBOUNDNAT 
which solves a very common, but specific, problem. 
Suppose you have an internal host on a RFC1918 
address (unrouted) network and someone has decided 
to poke a hole in the firewall to let outsiders access it. 
The firewall will do some kind of “reverse NAT” so 
that packets destined to a particular public address will 
be re-written and sent to an internal host. While the 
author feels this has risky security implications, at 


www.example.com@EXTERNAL ANAMES=example.com@EXTERNAL 
ANAMES=vpn.example.com@EXTERNALONLY 


Internal DNS Records | 
[a |__| «| __r —__ 


External DNS Records 
A 


'database-examplecom | 6432.179.35 | database.cxamplecom || FAILS | d64-32-179-55 example.com | 
Fwonwexample.com | 64.32.179.56 | www.example.com || 64.32.179.56 | www.examplecom | 
exampe.com [6432.17.56 | wwwexample.com || 68.32.179.56 | wwrw.example.com | 
[-64.32.179.57 | vpn3000.example.com [| 64.32.179.57 
[FAILS | vpn3000.example.com [| 64.32.179.57 | vpn.example.com 
[raics [FAILS] 64.32.179.55 | dd-32-179-55.cxample.com_ 
| [essai 

i Ul 


FAILS FAILS 64.32.179.56 | vpn.example.com 
be 


Display 3: Demonstration of scopes. 













vpn3000.example.com 








vpn.example.com 
D64-32-179-55.example.com 
D64-32-179-56.example.com 
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least HostDB lets you get the DNS records correct 
when you choose to use this technique. Here’s how 
one would do this with HostDB: 


64.32.9.8 host.example.com@INBOUNDNAT 
10)..1 .1.5 host.example.com 


Scopes can pertain to each hostname listed on a 
line, whether it is the host, a CNAME or an ANAME. 
A host can have multiple ANAMES and CNAMES 
and each name can be normal, external, externalonly, or 
inboundnat independently of the others. The generation 
process becomes quite complicated. What is the right 
PTR record to generate if a host is externalonly but has 
an ANAME alias that is normal? (Our answer: the 
ANAME on the inside, the externalonly name on the 
outside). As new situation cropped up we had to fine- 
tune the algorithms. 

Mail Servers 

Marking a host as a mail server means the MX 
records should be slightly different. It should have the 
default MX records but also an MX record pointing to 
itself that is better priority than the others. You specify 
this in hostdb.txt with the ISMAILSERVER flag; see 
Display 4. 


096.165...112...13 mtal.example.com 
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HostDB does not create duplicates, even in 
tricky situations like when the default MX record for a 
subnet is the same as the host. For example, if 
zoneconf. txt specifies that all hosts are to receive MX 
records of priority 10 for mtal.example.com and pri- 
ority 20 for mta2.example.com and hostdb.txt contains 
the code in Display 5, then the example.com zone file 
would resemble Display 6. 


Host “normal” receives both MX records. “‘msex- 
change” receives the MX records and the best priority 
MX pointing to itself. “mtal” and “‘mta2” have the 
best priority pointing to itself with a worse priority MX 
pointing to either mta2 or mtal as appropriate. 

DHCP Pools 


HostDB provides assistance to anyone creating 
large DHCP pools. DHCP pools usually assign 
“generic” names to each IP address such as 
ool-24-12-12-12.comcast.net. Repetitive jobs should 
always be automated, thus there is a syntax for auto- 
matically generating such hostnames. A DHCP pool in 
the middle of a subnet might look like Display 7. 


This would generate a host name called 
dhcp-10-1-40-4-pool.example.com for the first DHCP_ 
POOL entry, and similar entries for the rest. 


ISMAILSERVER 


Display 4: Mail server specification via “ISMAILSERVER”. 


198.65.112.9 normal.example.com 


198.65.112.10 msexchange.example.com ISMAILSERVER 
198.65.112.11 mtal.example.com ISMAILSERVER 
193 :163.0. L212 mta2.example.com ISMAILSERVER 
Display 5: Mail servers with varying priorities in zoneconf.txt . 
normal IN A 198: (65 sb 12.09 
IN MX 10 mtal 
IN MX 20 mta2 
msexchange INA 198.65.112.10 
IN MX 0 msexchange 
IN MX 10 mtal 
IN MX 20 mta2 
mtal IN A 19865 2172,21 
IN MX 0 mtal 
IN MX 20 mta2 
mta2 IN A 198::65 212.12 
IN MX 0 mta2 
IN MX 10 mtal 
Display 6: Resulting example.com zone file. 
10.1.40.1 zathras-purple.example.com ISROUTER=zathras.example.com 
10.1.40.2 server2.example.com 
10.1.40.3 server3.example.com 
DHCP_POOL_TEMPLATE dhcp-$a-S$b-$c-$d-pool.example.com 
10.1.40.4 DHCP_POOL 
10.1.40.5 DHCP_POOL 
10.1.40.6 DHCP_POOL 
10.1.40.7 DHCP_POOL 
10.1.40.8 DHCP_POOL 
10.1.40.9 DHCP_POOL 
10.1.40.10 host10.example.com 
Display 7: DHCP pool in the middle of a subnet. 
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The template takes effect from the point in the file 
it appears and continues until another DHCP_POOL_ 
TEMPLATE line overrides it. For example, at one loca- 
tion they wanted to have slightly different hostnames 
for a DHCP pool for IP phones. In that address range, 
they put the lines shown in Display 8. 


There is no coordination between the DHCP_ 
POOL template and the DHCP configuration file that is 
generated. That is, if you use the DHCP_ POOL feature 
to generate a range of similarly-named systems, you 
must manually update the DHCP configuration file to 
include those ranges. 


A future enhancement would be to have the tem- 
plate be a stack, permitting one to more easily switch 
to a new template and switch back when one is done. 


While it might be useful to be able to specify a 
range of IP addresses (rather than to list one per line) 
this feature has been delayed until a clear syntax can 
be developed. A range would violate the “one line per 
address” rule that the sysadmins preferred. It turns out 
they often use their text editor’s command to cursor- 
down 256 times to jump from subnet to subnet. Thus, 
any command that consolidates repetitive lines makes 
navigation more difficult. 


However, a command-line tool is provided to 
generate lists of IP addresses with a template. The 
command was used to create the above range: 


genrange 10.1.100.4 10.1.100.100 \ 
’ $ip<tab>DHCP_POOL’ 


Zone Configuration 


Zoneconf.txt stores everything required to gener- 
ate the zones outside of the host information itself. 
Settings here change rarely and are usually the 
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purview of senior DNS engineers. By keeping these 
options separate we avoid accidental modifications by 
junior engineers. The lines of this file are executed in 
order, like a sequential programming language. 
The file usually starts by setting some options: 
## When a hAAA-BBB-CCC-DDD external name 


ft is created, what domain is it in? 
OBSCUREZONE example.com 


OBSCUREZONE indicates the domain name to use 
for obscured, or anonymous, hostnames in external 
zone files. 


SOA, MX, and ZONESERVERS sets the SOA 
values, default MX records, and NS records to be 
included in any zone file created after this point; see 
Display 9. 

Notice that most commands are repeated once for 
INTERNAL and once for EXTERNAL. These settings 
are different for internal and external zones. There is 
also ““SLAVE”, which is an internal secondary. There 
is no support for external slave servers at this time as 
they tend to be at ISPs and don’t let customers update 
their DNS server configurations directly. 


When the system generates the named.conf file it 
will need to know some parameters to insert into any 
““zone”’ setting. For example, we need to specify who 
is allowed to do zone transfers and who is allowed to 
do dynamic DNS updates. We set those values as 
shown in Display 10. 

Now the exciting part where we actually create 


some zone files. We do these with commands like 
DOMAIN and REVDOMAIN; see Display 11. 


These lines direct HostDB to generate the zone 
file for corp.example.com, then example.com, then 
ex.com. 


DHCP_POOL_TEMPLATE phone-Sa-S$b-S$c-S$d-pool.example.com 


10.1.100.4 DHCP_POOL 
..elided for space... 
10.1.100.100 DHCP_POOL 


DHCP_POOL_TEMPLATE dhcp-S$a-S$b-S$c-$d-pool.example.com 
Display 8: Example template for an IP phone pool. 


SOA INTERNAL hostmaster.example.com 1H 15M 30D 60M 
SOA EXTERNAL hostmaster.example.com 1H 15M 30D 60M 


MX INTERNAL 10 lucy.example.com 
MX EXTERNAL 10 exgw2.example.com 


20 ingw2.example.com 
10 exgw4.example.com 


ZONESERVERS INTERNAL lucy3.example.com ingw2.example.com 
ZONESERVERS EXTERNAL exgw4.example.com exgw2.example.com 


Display 9: SOA, MX, and ZONESERVERS examples. 


ALLOW-TRANSFER INTERNAL 
ALLOW-TRANSFER SLAVE 
ALLOW-TRANSFER EXTERNAL 


ALLOW- UPDATE INTERNAL "none" 
ALLOW- UPDATE SLAVE "none" 
ALLOW- UPDATE EXTERNAL "none" 


"“primary_servers" 
"primary_servers" 
"rev65_servers" 


Display 10: Zone transfer and dynamic DNS update permissions. 
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Order is important here. After a host’s DNS 
records are output they will not be output again. If 
example.com precede corp.example.com, the zone file 
for corp.example.com would be nearly empty, contain- 
ing only the required SOA and NS records. 


Notice that ex.com is only an external zonefile. 
That’s permitted. Domains can be internal-only too. If 
you mix and match HostDB will do the right thing. 
This can be useful for subdomains that exist only 
inside the company. 


Now we can generate the reverse lookup zone 
files; see Display 12. The REVDOMAIN options indi- 
cate what size in-addr.arpa zone file to create (class A, 
B, or C), the starting IP address, and whether this is 
internal only or both internal and external. 


There are a few tricks one can do. For example, 
since the configuration is “executed” line by line, you 
can change settings between zones. For example, if 
you had different MX records for a particular domain, 
you can change along the way; see Display 13. 


This would generate corp.example.com with lucy 
and ethel as the default MXs. Then the default MXs 
would changed to betty and wilma before prod.exam- 
ple.com is generated. Then the SOA defaults would be 
changed before the reverse domain 197.32.64.in- 
addr.arpa would be generated. 


Rudimentary support for RFC 2317-style “class- 
less” delegations is implemented but not complete. 
Eventually there will be a REVRANGE command that 
accepts a range like that shown in Display 14. This 
will generate a zone file with the specific contents 
required for RFC 2317. 


DOMAIN corp.example.com 
DOMAIN example.com 
DOMAIN ex.com 


INTERNAL 
INTERNAL 
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Special Cases 


While HostDB generates a file named.conf file it 
generates a very standard “zone {}” stanza for each 
DOMAIN command. If you need something very spe- 
cialized, one can use CUSTOMDOMAIN instead. 
This will generate no “zone {}” stanza and rely on 
you to add one in manually using other means 
explained later. 


This is particularly important for supporting 
Dynamic DNS updates. HostDB supports you by giv- 
ing you enough rope to hang yourself. 


Summary 


That’s all there is to it! You specify what zone 
files to create and it generates them plus the 
named.conf file. All the hard work is done for you. If 
extra lines need to be added to a zone file or configu- 
ration file, that can be done with the notation 
described in the next section. When the system is fully 
configured it should be able to generate all files with- 
out any manual intervention. 


Header and Footer Files 


HostDB can’t be all things to all people, so there 
is an “emergency escape hatch” that lets you cus- 
tomize any file that is generated. You can add a header 
file and footer file to any file created by HostDB. For 
any file X generated by HostDB, the file that is actu- 
ally created is made up of: 

1. The contents of file STEMPLATES/X-head (if 
this file exists). 

2. The content generated by the HostDB system. 

3. The contents of the file STEMPLATES/X-tail 

(if this file exists). 


EXTERNAL 
EXTERNAL 
EXTERNAL 


Display 11: Creating zone files using DOMAIN. 


REVDOMAIN CLASSC 64.32.197.0 INTERNAL 
REVDOMAIN CLASSC 64.32.198.0 INTERNAL 
REVDOMAIN CLASSC 10.1.0.0 INTERNAL 
REVDOMAIN CLASSC 10.1.255.0 INTERNAL 
REVDOMAIN CLASSB 10.100.0.0 INTERNAL 
REVDOMAIN CLASSC 10.254.0.0 INTERNAL 
REVDOMAIN CLASSC 172.17.17.0 INTERNAL 


EXTERNAL 
EXTERNAL 


Display 12: Creating reverse look up zone files using REVDOMAIN. 


SOA EXTERNAL hostmaster.example.com 1H 1H 30D 60M 


MX INTERNAL 10 lucy.example.com ; 
DOMAIN corp.example.com INTERNAL 


MX INTERNAL 10 betty.example.com 
DOMAIN prod.example.com INTERNAL 


20 ethel.example.com 
EXTERNAL 


20 wilma.example.com 
EXTERNAL 


SOA EXTERNAL hostmaster.example.com 2H 2H 30D 60M 


REVDOMAIN CLASSC 64.32.197.0 


INTERNAL EXTERNAL 


Display 13: Specifying varied MX records for a particular domain. 


REVRANGE 22.33.44.32 22.33.44.63 32-63.44.33.22.in-addr.arpa INTERNAL EXTERNAL 
Display 14: Future style of RFC 2317-style range specification. 
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(The file name suffixes “head” and “‘tail” are 
used instead of “footer” and “header” because they 
sort better that way.) 


This feature is useful for: 

e Sneaking special records into a zonefile. If 
HostDB won’t generate a zone the way you 
want it, add the lines to a -tail file. 

e Adding custom zones to a named.conf file. If 
there are zones not under HostDB management 
you can still add information about them in 
named.conf 

e Coarsely approximating an “‘include file” 
mechanism. If a DHCP server configuration 
file doesn’t have an “include”? mechanism, you 
can wrap the contents of a file around the out- 
put of HostDB’s dhepd.conf 


DHCP Generation 


HostDB parses the hostdb.txt file and creates the 
“host” statements appropriate for ISC’s DHCP server. 
One stanza is created for each host that has a MAC= 
setting. No stanzas are created for hosts without 
MACS settings. 


A hostdb.txt line that looks like Display 15 
would generate the default template like that shown in 
Display 16. This is appropriate for most systems. 
(Note that the MAC address is reformatted to be pretty 
by capitalizing the letters and zero-padding.) 


However, if TYPE=foo is included, as alternate 
template is used. For example we might want a differ- 
ent template if we know the machine has a broken 
DHCP client that gets confused by extra data. A 
hostdb.txt line like this would select the template 
called ‘“‘win95” which is intentionally minimal (see 
Display 17) generates the code shown in Display 18. 
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If the template includes a hyphen, the text after 
the hyphen is passed to the template as a parameter. 
Thus templates can be smart. Suppose there is a 
default netboot server and a special one for develop- 
ers. Templates can be constructed that produce proper 
DHCP configuration for each; see Display 19. 


Currently the templates are written as subrou- 
tines in Perl and are hard coded into the mkdhcp 
script. Future versions will move these templates to 
their own files, though this work has been deferred to 
later because (1) the need for special templates to han- 
dle broken DHCP clients is reduced over time as more 
vendors figure out how to write DHCP clients that 
aren’t broken, (2) nobody’s really complained about 
the system as it is. 


Deployment (Phear My Aw4z0m3 Makefile Skilz!) 


Generating zone files and configuration files is 
not enough. They must be deployed to be useful. 


Rather than generate files directly where they 
will be used, HostDB generates all files in a directory 
called ‘‘out” (short for output). Sites with a very large 
legacy system for DNS/DHCP management can sim- 
ply copy the newly generated files (or just the files 
that are useful to them) out of this directory. However, 
HostDB includes a tool that makes this even easier 
called mkdestination. It generates a Makefile that 
“does the right thing.” 


Mkdestination takes ‘“‘destination.txt” for input, 
and generates “destination.mk” which can be used in 
the Makefile that drives your DNS/DHCP service. 

Consider short example destination.txt file 
shown in Display 20. The format is a simple list 
source files, the “*->’’, then destinations. Destinations 


10.10.244.14 dell14.example.com MAC=00:5:3b:F1:21:7a 
Display 15: Sample hostdb.txt file. 


host dell4.example.com { 


hardware ethernet 00:05:3B:F1:21:7A; 


fixed-address 10.10.244.14; 


option host-name "dell4.example.com"; 


ddns-hostname "DELL4"; 


Display 16: Generated default template from hostdb.txt file above. 


10.10.244.14 dell4.example.com MAC=00:5:3b:F1:21:7a TYPE=win95 
Display 17: Minimal template for ““win95”’. 





host dell4.example.com { 


hardware ethernet 00:05:3B:F1:21:7A; 


fixed-address 10.10.244.14; 


Display 18: Code generated by template above. 


10.10.244.13 humanl.example.com MAC=00:05:3B:F1:21:7A TYPE=netboot 
10.10.244.14 human2.example.com MAC=00:05:3C:E3:32:A4 TYPE=netboot-main 
10.10.244.15 human3.example.com MAC=00:05:32:A2:01:12 TYPE=netboot-dev 


Display 19: DHCP configuration templates. 
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can be a file name or directory on the local machine or 
some other. There is no macro expansion or other sub- 
stitution mechanism because we want to keep the for- 
mat simple. 


The first line says that to deploy INTER- 
NAL.example.com and INTERNAL.198.32.64.in-addr. 
arpa one should copy it into the local directory 
/var/named, and scp it to a /var/named on a machine 
called “betty”. The next two lines specify that a file is 
to be copied to a particular filename. This permits one 
to rename files as they are copied. 


Display 21 shows how one might use mkdestina- 
tions in a master Makefile. 


Invoking “make push” would generate the desti- 
nations.mk file (if needed) then run the “all” recipe. 
The recipes in the destinations.mk file “do the right 
thing” for all files mentioned in destinations.txt. 
“make push-local” is similar but runs the “all-local” 
recipe, which only deploys files destined for the local 
machine. This is useful for testing purposes. 


Mkdestinations is needed to overcome limita- 
tions in ““make’’. “‘Make”’ has no awareness of remote 
file timestamps, nor can “make” handle a dynamic 
list of zones and files with multiple destinations. 


Efficient Updates of Remote Files 


Make has no idea if betty:/var/named/named. 
conf is up to date because it is on a different machine. 
Therefore, mkdestination compensates by creating 
local timestamp files after a remote operation is com- 
plete (that is, after a remote file is updated success- 
fully it touches a local file whose name encapsulates 
the destination server and filename). Future timestamp 
comparisons can be done against the local file. Rather 
than recopying a file to a remote machine “‘just in 
case’’, the timestamps can be compared. In the case of 
copying a file to betty:/var/named/named.conf, the 
timestamp file ds/ betty__var_named_named.conf is 
created. (The actual filename is much more compli- 
cated to encode special characters that might cause 
problems.) 


Mkdestination also creates a recipe called 
“clean”’ which deletes all timestamps. Thus, to force a 
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“push” to all other hosts, simply “make clean” then 
“make push” and all files will be copied whether they 
need it or not. 


Efficient Zone Serial Number Updates 


The other interesting feature of mkdestination is 
that it is smart about zone serial numbers. 


DNS zone files have an embedded serial number 
which is used to determine when zone transfers are to 
happen. If a secondary has a zone of serial number X, 
and is told that the primary has a higher serial number, 
it requests a download of that zone (a “zone trans- 
fer’). Zone transfers are a CPU- and (possibly) net- 
work-intensive operation. We want to prevent gratu- 
itous zone transfers. 


The problem, however, is that the only way to 
know if a zone file will change is to generate the new 
one and compare it to the old one. The comparison 
can’t be a simple “cmp” or “diff” because the serial 
number inserted into the zone file will be different 
every time. Most changes to hostdb.txt tend to affect 
only a few zone files. 


Again, mkdestination to the rescue. mkdestina- 
tion knows about serial numbers and can do compar- 
isons that ignore the serial number line of a zonefile. 
The methodology is as follows. 


HostDB generates all files in a directory called 
out”. Any file that needs a serial number contains 
the text ‘‘:serial:” anywhere a serial number is to be 
placed. This makes comparison of zone files easier. 
Any file that did change is copied to a directory called 
MP and another copy is placed in a directory called 
MS. The MP directory contains a plain copy of the 
zone but the MS directory contains a copy of the file 
with ‘“:serial:” changed to the serial number used for 
all files generated at that time. When deciding which 
files need to be deployed, the timestamp of MS is used 
to determine if the file needs to be copied but the file 
that is copied comes from MP. 


“ 


After a file is successfully deployed, a timestamp 
file is created in the DS directory marking the comple 
tion. In the case of copying a file to betty:/var/named 
named.conf, the timestamp file with a name like DS/ 
betty__var_named_named.conf is created. The result is 


INTERNAL.example.com INTERNAL.198.32.64.in.addr.arpa -> \ 
/var/named/. secondary:/var/named/. 


INTERNAL.named.conf -> /etc/named/named.conf 


SLAVE.named.conf -> betty:/etc/named/named.conf 


Display 20: Sample destination.txt file. 


destinations.mk: ../destinations.txt 


mkdestinations <../destinations.txt >destinations.mk 


push: destinations.mk 
make -f destinations.mk all 


push-local: destinations.mk 
make -f destinations.mk all-local 


Display 21: Using mkdestinations in a Makefile. 
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that files are only copied when absolutely needed. It is 
optimal. 


At a site with dozens of zones the DNS servers 
were being hit fairly hard by all the zone transfers. 
After mkdestination’s technique was deployed the 
administrators were very happy to see that very few 
zones were actually being transferred after typical 
updates. It was quite impressive to see many, many 
zone files with an assortment of serial numbers as 
diverse as the history of updates had dictated. Yet, no 
tool more complicated than ‘“mkdestination” and 
“make” was needed. 


If this sounds confusing to you, just be happy in 
the knowledge that you don’t have to understand it to 
benefit from the efficiencies. Files that don’t need to 
be copied across your network aren’t copied. 


Deployment Examples 


Here are two examples of how to deploy 
HostDB. First we will consider a new site starting 
from scratch. Then we will see how easy it is to 
deploy in a pre-existing environment. 


We need a place to put our files. The HostDB 
files can be put in any directory as your site’s require- 
ments dictates (/var/hostdb or /home/adm/hostdb is 
typical). HostDB assumes that all executables will be 
found in the shell’s PATH. As a result, test datasets 
can be placed anywhere (even /tmp). When testing 
new executables, one can simply put the newer ver- 
sions ahead of the older ones in the shell’s PATH. 


In these examples we will use /var/hostdb and 
assume the PATH is set correctly. 


LOi7..1..:0 
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Example 1: Starting from Scratch 


In this example we are starting from scratch. 
Imagine we are setting up a new network for a new 
company. We have one external network (64.32.179.0/ 
24) and one internal network (10.1.1.0/24). There will 
be a dhcp pool from .140 to .250 in the first network. 


We begin by generating the hostdb.txt file. We 
generate a list of each IP address. That way when a 
junior engineer goes to allocate an IP address, they 
don’t have to understand anything but (1) find an 
unused address on the right subnet, (2) remove the 
comment symbol and change the hostname as appro- 
priate. This prevents them from introducing typos into 
the list of IP addresses, and prevents mistakes such as 
thinking they can create new IP subnets by simply 
editing this file. It also keeps the file neat and orderly 
in case you work with people that can not be trusted to 
keep a file in numerical order. 


Display 22 shows commands that will generate a 
hostdb.txt that is a good starting point. 
e Line | creates commented-out sample lines for 
the first 140 addresses. 
e Line 2 creates addresses for our DHCP pool. 
Note the custom template. 
e Line 3 completes the internal network’s addresses. 
e Line 4 uses a different template for the external 
hosts. It also demonstrates that if the second IP 
address is replaced by an integer, it is inter- 
preted as a count of how many lines to output 
instead of an end address. 
Display 23 shows the output. 


Now edit the hostdb.txt to include the initial 
hosts that will be installed. To make sure that nobody 


TOals 1 «ISS 


10.1.1.140 10.1.1.250 *S$ip\tDHCP_POOL’ 
1G601,1,257 Wal .1 95 
64.32.179.0 256 ‘#Sip\tex$hex.$DOMAIN\@EXTERNAL’ 


Display 22: Generating hostdb.txt from scratch. 


genrange >hostdb.txt -d example.com 
genrange >>hostdb.txt -d example.com 
genrange >>hostdb.txt -d example.com 
genrange >>hostdb.txt -d example.com 
#10.1.1.0 UNUSEDOA010100.example. 
#10.1). 12 UNUSEDOA010101.example. 
...@lided for space... 
#10.1.1.138 UNUSEDOA01018A. example. 
#10.1.1.139 UNUSEDOA01018B.example. 
LOD. Lt 40 DHCP_POOL 
10.1.1.141 DHCP_POOL 

..elided for space... 
10.1.1.249 DHCP_POOL 
L0.4:; 16250 DHCP_POOL 
#10.1.1.251 UNUSEDOA0101FB.example. 
#10.1.1.252 UNUSEDOAO101FC. example. 
#10.1.1.253 UNUSEDOAO101FD. example. 
#10.1.1.254 UNUSEDOAO101FE. example. 
#10.1.1.255 UNUSEDOAO101FF.example. 
64.32.179.0 ex4020B300.example.com 
64.32.179.1 ex4020B301.example.com 
...elided for space... 
64.32.179.254 ex4020B3FE.example.com 
64. 32:.179...255 ex4020B3FF.example.com 


com 
com 


com 
com 


com 
com 
com 
com 
com 


Display 23: Output from above commands. 
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tries to allocate a broadcast address, we mark the “‘all 
zeros” address as “‘foo-net.example.com”’ and the “‘all 
ones” address as “‘foo-bcast.example.com” where 
“foo” is a unique name for each subnet. Display 24 
shows the non-comment lines. 


Display 25 shows our zoneconf.txt file, which 
is almost the same as the example provided with the 
software. Most sites can use this template and simple 
change ‘“‘example.com”’ to their domain and edit 
their MX records and ZONESERVERS to suit their 
needs. 


The last three lines show which zones to generate. 
First the zone for example.com, which has different 


1Qi..1..d..0 main-net.example.com 
1001 tak zathras-main.example.com ISROUTER=zathras.example.com 
10'..1 1.2 fileserver.example.com 
10), hed mailserver.example.com ISMAILSERVER 
10.1.1.4 vector.example.com 
LOD (BO staffpcl.example.com MAC=00:b0:d0:a6:cf:fl 
LO all Slrabel staffpc2.example.com MAC=00:b0:dl:a7:c0O:dl 
LO: 5 WAHL staffpc2.example.com MAC=00:b0:c2:b3:c3:d3 TYPE=freebsd 
10.1.1.140 DHCP_POOL 
10\.151.041 DHCP_POOL 
10.1.1.142 DHCP_POOL 
...elided for space... 
10.1.1.249 DHCP_POOL 
20:.'1 .1..250 DHCP_POOL 
LO.d .1.. 255 main-bcast.example.com@EXTERNAL 
64.32.179.0 ext-net.example.com@EXTERNAL 
64.32.179.1 isp-router.example.com@EXTERNAL 
64.32.179.2 zathras-ext.example.com@EXTERNAL ISROUTER=zathras.example.com@EXTERNAL 
64.32 179.3 mailqueue.example.com@EXTERNAL ISMAILSERVER 
64.32.179.4 vector.example.com@INBOUNDNAT 
64.32.179.5 exweb.example.com ANAME=www.example.com@EXTERNAL 
64.32.179.255 ext-bcast.example.com@EXTERNAL 
Display 24: Edited hostdb.txt file. 
TEMPLATEDIR 


OBSCUREZONE example.com 
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values for the internal and external version. Next we 
generate the reverse lookup zone for 10.1.1.0/24, 
which only generates an internal zone file. Lastly we 
generate the 64.32.179.0/24 zone reverse lookup 
which has different internal and external values. 


Copy the template Makefile into the main direc- 
tory and we are ready for our first attempt at generat- 
ing our zones. To save ourselves typing, we’ll create 
an alias for bash/sh/ksh or csh/tcsh (see Display 26). 
Then we can do “makeh” to generate a new set of 
files. If we are happy we can do “makeh push” to 
deploy the files. As a result, the “out” directory now 
contains: 


SOA INTERNAL hostmaster.example.com 3h 1h 1w ih 
SOA EXTERNAL hostmaster.example.com 3h 1h 1w ih 


MX EXTERNAL 10 mailserver.example.com 
MX INTERNAL 10 mailqueue.example.com 


20 sl.isp.com 


ZONESERVERS INTERNAL fileserver.example.com mailserver.example.com 
ZONESERVERS EXTERNAL mailqueue.example.com exweb.example.com 


ALLOW- UPDATE INTERNAL done 
ALLOW- UPDATE INTERNAL none 
ALLOW - UPDATE SLAVES none 
ALLOW- UPDATE EXTERNAL none 


DOMAIN example.com 
REVDOMAIN CLASSC 10.1.1.0 


INTERNAL EXTERNAL 
INTERNAL 


REVDOMAIN CLASSC 64.32.179.0 INTERNAL EXTERNAL 
Display 25: Example zone.txt file. 





## bash/sh/ksh alias: 


alias makeh=’cd /var/hostdb/out && make 


## csh/tcsh alias 


alias makeh ’cd /var/hostdb/out && make 


../Makefile’ 


../Makefile’ 


Display 26: Shell ‘makeh’ alias definitions. 
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$ ls -1 
EXTERNAL.179.32.64.in-addr.arpa 
EXTERNAL. example.com 

EXTERNAL. named.conf 

EXTERNAL. named. root 
INTERNAL.1.1.10.in-addr.arpa 
INTERNAL.179.32.64.in-addr.arpa 
INTERNAL.example.com 
INTERNAL.named.conf 
SLAVE.named.conf 


Notice that each filename is prefixed with “INTER- 
NAL.”, “EXTERNAL.” or “SLAVE.”. All other 
files are tagged as being appropriate for either the 
inside or the outside. EXTERNAL.named.root is the 
NIC’s “‘root cache”’ file which the Makefile automati- 
cally retrieves via FTP. 


Now check the zone files by manual inspection. 
Correct any errors by editing ../hostdb.txt and re-run- 
ning ““makeh” until you are satisfied. 


Notice that in addition to zone files, ISC BIND 
named.conf files are generated. If you examine them, 
you'll find them very anemic. They just contain the 
“zone {}” entries which are automatically generated. To 
create a complete file, determine what you would put 
before and after the generated parts and put them in the 
proper -head and -tail files. Any text in ./INTER- 
NAL.named.conf-head is prepended to the generated 
file, and any text in ../[NTERNAL.named.conf-tail is 
appended to the end of the generated file. The same is 
true for all files generated by the system. 


Now that we are happy with the files being gen- 
erated, it’s time to tell HostDB where to put them. To 
do that, we create ../destinations.txt which lists where 
each file is to be copied; see Display 27. 

e Line | specifies that EXTERNAL.named.conf 
is to be copied to mailqueue.example.com in a 
directly called /etc/namedb. It is renamed to 
named.conf as it is copied. It is also copied to 
exweb. 
Line 2 specifies three files that are to be copied 
to two servers in their /var/named directories. 
The files are not renamed as they are copied. 
Instead, we will make sure the configuration 
files that refer to these files must specify the 
filename as they exist. 
e Line 3 is like line | but for the internal servers. 
e Line 4 is analogous to line 2. 
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This command will generate a makefile that will do the 
actual copying: makeh destinations.mk. Finally, 
“makeh push” will actually copy the files into place. A 
few rounds of debugging and the system is deployed. 


Example 2: Replacing a Legacy System 


In this scenario we have a working system where 
we hand-edit zone files and copy them using a shell 
script. To move from a legacy system to HostDB 
requires a lot of testing. HostDB includes utilities that 
help every step. 


The first and most difficult step is to convert the 
zone files and turn them into hostdb.txt format. This is 
the reverse of what HostDB does! Sadly, we can’t 
reverse the polarity of the flux capacitor and have 
everything “just work.”’ Luckily, the HostDB package 
includes zone2hostdb which does 90% of the work for 
you. It takes a zone file as input: 


zone2hostdb <zonefile >hostdb.txt-base 


Since this step is automated we reduce the poten- 
tial for mistakes considerably. 


This gives you a first draft of what hostdb.txt 

should be. Now check these issues: 

e Verify all mail servers are listed as ISMAIL- 

SERVER 

¢ Verify that all routers and multihomed hosts are 
marked as ISROUTER or ISMULTIHOMED 
as appropriate 
Mark any broadcast addresses so they are not 
accidentally used 
Check any ANAMEs to verify that the official 
name is listed first on the line and that actual 
aliases are listed in an ANAME. This assures 
proper PTR records 
If any special cases are in place for PTR records, 
make sure the name to be used for the PTR 
record is always the first hostname on the line 
Set the scope to @EXTERNAL for any hosts 
accessible externally 
Set the scope to @INBOUNDNAT for any host 
that is accessed by external users by a different 
address than internal users 
e Set the scope to @EXTERNALONLY for any 

hostnames only external users should access 

(very rare) 
e Anything else “special” about your DNS zones 


EXTERNAL.named.conf -> mailqueue:/etc/namedb/named.conf  \ 
exweb:/etc/namedb/named.conf 


EXTERNAL.example.com EXTERNAL.179.32.64.in-addr.arpa \ 


EXTERNAL.named.root -> \ 


mailqueue:/var/named/. exweb:/var/named/. 


INTERNAL.named.conf -> fileserver:/etc/namedb/named.conf \ 
mailserver:/etc/namedb/named.conf 


INTERNAL.example.com INTERNAL.1.1.10.in-addr.arpa \ 
INTERNAL.179.32.64.in-addr.arpa -> fileserver:/var/named/. \ 


mailserver:/var/named/. 


Display 27: File ../destinations.txt: where files are to be copied. 
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The next step is to use genrange to enumerate 
every IP address that should be listed in your 
hostdb.txt file, whether it is commented out or not. We 
use the same ‘“‘genrange” commands as in the first 
example except we save the output to a file called 
hostdb.txt-enum. 


HostDB includes a utility called “mergeiplists” 
that will merge these two lists properly even though 
some of the lines are commented out. It assumes the 
first appearance of an IP address is the authoritative 
line and throws the others away. Thus, we list the 
hostdb.txt-base first because our hand-edited file 
should be authoritative; hostdb.txt-enum is just to fill 
in the gaps; see Display 29. Now we have a pretty 
decent draft of our hostdb.txt file. 


Before we can put it into use, we should test it 
extensively and then only switch to it incrementally. 


Testing is made easier by a utility that is included 
called ‘‘canonzone’’. Comparing two zone files can be 
difficult because of small changes in white space, sort- 
ing, and so on. “‘canonzone”’ reads a zone file and out- 
puts it in a very specific, clean, regular, format (a 
canonical format). If you are comparing a legacy zone 
file and a generated zone file, pass both thought 
canonzone first and the comparison will be much eas- 
ier. The distribution includes a file called ““Exam- 
ple_comparezones” which takes two files, passes 
them through canonzone, strips them down to just 
DNS A records, and compares them. 


Here are some tips for testing your new zones: 
¢ Use Example_comparezones as a template for 
your own comparison scripts. 

e Review each email server and verify the MX 
records are as required. 

e Review all other servers and verify their DNS 
records are as expected. 

¢ Review each external host and verify all of its 
records (test from inside and outside). 

e Set up a new DNS server and load the zones. 
Make sure they load without outputting any 
errors or warnings in BIND’s logfile. 

® Query the DNS server to make sure you get 
expected results for mail servers and other 
important hosts. When you are satisfied, ask 
co-workers to do the same. 

e When you are satisfied, configure yourself and 
a few trusted users’ machines to use this new 
DNS server. If anything breaks, fix it. After a 
few days, have your coworkers switch to this 
DNS server. Make sure they know to report 
problems to you. 


When you are satisfied with the zone files, you 
can slowly migrate them into your legacy system. The 
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destinations.txt file can be used to roll out just the spe- 
cific zones you are confident in. 


Suppose your legacy system loads your main 
zone from a file called /var/named/zone.example. 
com..zone, you can replace that file with the HostDB- 
generated file with a statement like that shown in Dis- 
play 28. Obviously you should make good backups 
before deploying any new zones. 


To push that zone out, ““makeh push”’. 


Test, test, test. Be ready to revert to the legacy 
zone if the problems you find are insurmountable. Fol- 
low any in-house procedures related to change windows 
and so on. For example, do not introduce new zones: 

¢ Right before you leave for vacation 

¢ The same week as the quarterly results or taxes 
are being prepared 

¢ The week of a major deadline for your company 

e The week raises and promotions are being 
decided 


Lastly, you can replace your named.conf files with 
the ones that are generated by HostDB (though if your 
zone filenames change, be very careful!). You should 
also integrate the Makefile into your system so that it 
controls all DNS-related activity from one central point. 


Related Work 


There are many other open source systems that 
do some or part of what HostDB does. There are even 
a few that do a lot more, especially where dynamic 
registration is involved. However, the goal of HostDB 
is to maximize features without having to resort to 
requiring a web front-end or SQL backend. The author 
believes HostDB is superior to all systems that don’t 
require an SQL backend, and easier and faster to install 
than any of the systems that require SQL databases. 


Here is my comparison to some popular utilities: 

e h2n — O’Reilly’s ““DNS and BIND” by Paul 
Albitz and Cricket Liu comes with h2n (and an 
enhanced version is called h2n-hp). These are 
fine DNS generator scripts. While it can be 
used for one-time conversion, there are features 
that let you maintain the host list as a /etc/hosts- 
like file. However, this is a “kit”, not a com- 
plete system. It does not include anything like 
mkdestinations for deployment. No DHCP sup- 
port. Source: www.oreilly.com/catalog/dns3 . 

¢ DNSDusty — While this system does automate 
deployment of the files it generates, it requires 
a web interface (but no SQL database). It 
makes adding a host very easy for a junior 
sysadmin once it is set up. No DHCP support. 
Source: www.poochiereds.net/dnsdusty . 


INTERNAL.example.com -> dnsserver:/var/named/zone.example.com..zone 


Display 28: Replacing a legacy zone file. 





mergeiplists hostdb.txt-base hostdb.txt-enum >hostdb.txt 


Display 29: Merging lists. 
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e dnscvsutil — Maintains your DNS zone files under 
CVS control and automatically updates reverse 
zones. Helping someone do a better job of main- 
taining hand-edited zone files is exactly the oppo- 
site of what I want to encourage. Though, revi- 
sion control is better than no revision control. No 
DHCP support. Source: freshmeat.net/projects/ 
dnscvsutil . 

Sauron — Sauron is a free DNS/DHCP manage- 
ment system with Web and command line inter- 
faces. It can manage multiple servers and gen- 
erates complete dhcpd and named configura- 
tions from the database. This is a very complete 
and impressive package. It manages deploy- 
ment of generated files. However, it does 
require a web interface and SQL database. 
Source: http://sauron.jyu.fi/. 

Updatehosts — A very comprehensive system for 
managing DNS information. This is popular at 
many large commercial sites. However, the 
learning curve seems overly steep as it requires 
understanding of database relations. This is not 
for the neophyte or someone trying to set up a 
new site quickly. Requires SQL. No DHCP sup- 
port. Source: ftp://ftp.tic.com/pub/updatehosts . 


The Future 


New features are added to HostDB on a regular 
basis. These projects are being considered: 

e Make testing even easier: mkdestination should 
create a “diff” recipe that diffs the last distrib- 
uted against the most recently generated files. 
Eliminate the need to modify the default Make- 
file: Move ‘restart commands” to the mkdesti- 
nation system. 

Add classless delegation support: Complete sup- 
port for RFC 2317-style “classless” delegations. 
Add [IPv6 support: Add support for IPv6 
records. (Those long addresses aren’t going to 
be fun to type. We need some human-factors 
help here.) 

Improve legacy conversion: Update zone2hostdb 
to read multiple zones, including reverse DNS 
zones, and use the information to do a better job. 
In particular, PTR records could be an indicator 
of which name is a canonical name and which is 
an ANAME. 


Conclusion 


HostDB generates really cool DNS zone files with 
all the features required by sites with complicated DNS 
configurations, such as aliases, “external only” names, 
“host hiding” and multihomed hosts. It also generates 
the ISC BIND configuration files required for primaries, 
secondaries, and external DNS servers. DHCP configu- 
ration for static leases are generated for ISC DHCP 
using a very flexible template-based mechanism. It is 
easy to configure and deploy, yet sophisticated behind 
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the scenes. It does not offer complicated features such as 
real-time host authorization, web front-ends, or any fea- 
ture that required sophisticated and difficult to maintain 
databases. 


Deploying the files is made easier through the 
use of a simple Makefile that is generated for you. The 
updates are very sophisticated, doing the minimal 
number of updates locally and to remote machines, 
even being careful not to create “zone transfer 
storms” by not updating a new serial number for a 
zone that does not warrant it. 


Availability 


The software open source and is available at 
http://www.everythingsysadmin.com/hostdb on the web. 
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ABSTRACT 


Application uptime is critical to every administrator. The factors which cause system 
downtime are often handled by the operating system, but causes for application faults (e.g., software 
bugs, hardware faults, or human errors) are not addressed by standard system software. Recovery is 
left to humans, who may often compound the problem due to misdiagnosis or simple error. While 
availability issues have traditionally been addressed by expensive high-availability clustering 
solutions, the increasing complexity of software stacks requires a solution for all systems. 


In addition to the challenges of managing availability of higher level software, the modern 
operating system itself is composed of many interdependent software entities. A failure in any one 
of these components often cascades, causing failures in other components. A complex software 
model with many interdependent elements makes diagnosing failures very challenging for system 
administrators. The traditional init.d script mechanisms for UNIX are only a weak reflection of the 
intricate dependency relationships which exist on every system. 


We introduce the Service Management Facility (SMF) as a comprehensive way to describe, 
execute, and manage software services. SMF promotes the service to a first-class operating system 
entity, without requiring modification of application binaries or changes to the UNIX process 
model. It relieves the administrator from duties of application failure detection and restart, and 


provides sophisticated diagnosis tools when automatic repair is impossible. 


Introduction 


As software running on a system becomes more 
complex, the traditional separation of management of 
system startup from run-time management becomes 
untenable. The reliability of modern hardware, coupled 
with the ever-increasing complexity of modern software 
means that applications are just as likely, if not more 
likely, to be the point of failure on a system. Reboots 
are costly, especially on sophisticated hardware. 


A system administrator is expected to ensure that 
critical applications are always running. This is some- 
times done by manual monitoring or an expensive to 
maintain home-grown tool, or the administrator is 
expected to turn to software not well integrated with the 
operating system — either sophisticated monitoring pack- 
ages, or expensive and often complex high-availability 
clustering software. The unbundled solutions usually 
also focus on higher level applications, and can neglect 
potential failures of core operating system daemons. 


But, even higher level applications do not stand 
on their own. In order to provide business-critical 
functionality, often two or more pieces of software 
must be working and cooperating closely together. 
Failure of even a less critical application can have 
unforeseen consequences in important software. 

In addition to basic availability concerns for soft- 


ware services, the service management model needs to 
be significantly enhanced. Without a common model and 


interface, it is very difficult to ask the system even basic 
gn 


questions such as: “‘what’s running?”, “what’s broken?”, 
and ‘what applications are available on this system?”. 


Application management is a fundamental task 
of all administrators: managing applications should 
not require a bolt-on solution. Driving the service 
management interface into the operating system also 
brings significant benefits through encouraging evolu- 
tion of core operating system functionality to support 
the service, and creating a truly common interface for 
all user software: operating system daemons and third 
party products alike. 


The UNIX service management model has not 
evolved significantly beyond the traditional process 
model coupled loosely with service startup scripts, and 
is stretched in trying to meet modern system adminis- 
tration demands. The lack of functionality is costly for 
administrators, who must spend significant effort 
monitoring and managing systems that provide no fun- 
damental abstractions to allow this. 


SMF directly addresses all of these administra- 
tion gaps and provides an integrated solution for ser- 
vice delivery and management on UNIX systems. 
SMF defines a fundamental service model which is 
used to control system startup, introduces a manage- 
ment interface for system and application services, 
and delivers diagnosis and restart capabilities for all 
services to maximize application availability. 
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Background and Related Work 


The traditional System V and BSD style init.d/rc 
systems provide an extremely flexible mechanism for 
initiating arbitrary processes during system startup. 
For all of the flexibility provided, a number of defi- 
ciencies are apparent. We’ll enumerate these deficien- 
cies in terms of the System V implementation, but 
analogous problems exist in the BSD style system. 

© No systematic way to list services. As the init.d 
system does not even encourage, much less 
enforce a 1:1 mapping of system services to 
init.d scripts, a simple Is of rc directories is never 
sufficient to get a full list of services provided 
by the system. The services required to start 
core operating system components are often the 
most opaque to the administrator. 

¢ No persistent mechanism to indicate to the sys- 
tem whether a service should be running or not. 
Upgrade of software components often updates 
or even re-creates the corresponding Ic script, 
overwriting administrative customizations such 
as attempts to disable the service. 

¢ No formal dependency management interfaces. 
System components are usually very sensitive 
to ordering changes. The System V _ lexico- 
graphical ordering provides some latitude for 
the third-party service. Service developers can 
have their script run at a certain point during 
startup (before some services and after others) 
by giving the script an appropriate name. But, 
the lack of explicit dependency specification 
for a service makes modifications to the order- 
ing of core system services fragile, and leads to 
unexpected behaviour. 

¢ Incomplete manual restart capabilities. The 
individual init.d scripts are not necessarily 
stand-alone or idempotent. The administrator is 
not expected to execute all of the init.d scripts 
manually, so there is no simple way to restart 
many individual services without understanding 
the intricacies of their implementation. Closely 
cooperating services compound the problem; 
restart of one may require restart of its closely 
coupled dependent services. The init.d system 
provides no mechanism to determine service 
interdependencies, so significant administrative 
expertise is required to restart individual ser- 
vices without a reboot. 

e Nonexistent automatic restart capabilities. The 
init.d system does not include automatic detec- 
tion and restart of failed components. Detection 
and restart is left to orthogonal mechanisms 
such as inittab and inetd.conf. 

e Lack of correlation between processes and ser- 
vices. Services are becoming _ increasingly 
sophisticated; the traditional UNIX model of a 
single daemon process as the entirety of a ser- 
vice is no longer adequate. Multiple processes 
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are correlated into service boundaries by admin- 
istrative expertise rather than being programmat- 
ically represented as a first-class object. The 
UNIX process model complicates the analysis — 
daemons are often reparented to init and lose 
their parent-child association in the process tree. 

¢ Ticking configuration time bombs. There is no 
formal association between the current status of 
the system and the expected state when the sys- 
tem is restarted because starting a service has 
no impact on its settings on reboot. In addition, 
there is no systematic way to determine current 
versus configured state, so a reboot often leads 
to unpleasant surprises due to un-configured or 
mis-configured services. 


For rigidly specified service models, such as 
inetd, the deficiencies can be more easily addressed. 
However, a complete system management architecture 
must provide a model sufficiently rich for all system 
services. SMF addresses each of these deficiencies for 
general software services. 


Previous work in this area aims primarily to 
improve deficiencies of the init.d/rc system in system 
startup and manual service enable, disable and restart, up 
to and including a more formal statement of dependen- 
cies [1, 2, 3]. This work helps administrators gain better 
visibility into system service configuration and provide 
slight enhancements to administrative tools. However, if 
application availability is the crucial measurement, these 
sorts of enhancements only aid insofar as they slightly 
reduce the chances of administrative error. 


Some recent efforts [4, 5] are more sophisticated 
and include simple service restart. The service can be 
restarted either when requests for that service come in, 
or restarted when the service exits unexpectedly. How- 
ever, these efforts assume a single process is the pri- 
mary provider of service, which is insufficient for a 
large class of business-critical applications. 


Design Principles 


Altering system service management is guaran- 
teed to cause a change in administrative procedures. 
Changing something as fundamental as the init.d system 
requires administrators to spend time learning the new 
management paradigm. Thus, we took an ambitious 
approach so that the significant changes happened all 
together, rather than creating an incremental education 
cost over many software releases. The redesign of ser- 
vice management procedures must be comprehensive, 
and consider the full range of administrative proce- 
dures, software errors, and hardware errors. 


The most fundamental requirement of this work is 
a simple, common model for all services. Services must 
be able to encompass nearly all capabilities of the system 
— simple daemons, network services, complex databases, 
and even non-process-based services such as dump 
device configuration. The management framework must 
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be flexible enough to allow common actions to be done 
easily and with no required administrative knowledge of 
the service implementation. A common model for all 
application and system services is required in order to 
thoroughly leverage administrator expertise; an admin- 
istrator should be able to manage syslogd using the same 
basic commands as telnetd. 


The service model must be extensible. New 
types of services, increased diagnosability, and 
expanded meta-configuration should be easy to sup- 
port with no required enhancements to existing service 
descriptions. 

Relationships between services are more com- 
plex than start time requirements. Many services are 
written carefully to be resilient to failure of their 
dependencies. For example, a well written networking 
application does not need to be restarted when a tran- 
sient nameservice failure occurs. However, starting 
even a defensively-coded application without name- 
service availability is usually futile. Thus, a rich 
dependency declaration is required to encompass both 
startup and failure scenarios. 


Services then are restarted according to their 
dependencies following either hardware or software 
failure. A service restart must be attempted following 
any type of failure, such as a critical process exit, an 
uncorrectable memory error or other hardware error, a 
software coredump or premature exit, or an admin- 
istrator accidentally killing the wrong process. Any 
dependents of that service must be restarted if they 
specify intolerance to dependency failures. 


When a service cannot be automatically repaired 
after a fault, the system must provide significant aids to 
manual diagnosis, and trace failure back to the respon- 
sible component. When software failures can be traced 
back to the initial fault, this information as well as as 
much data as possible about the specific fault must be 
provided to the administrator so that he can spend his 
time repairing the fault rather than debugging an error 
in a dependent service. For example, if an application 
cannot start because the filesystem containing the 
application data was unable to mount, the system 
should point to the filesystem as the root cause of the 
fault, rather than incorrectly indicting the application. 


The other critical component of a service man- 
agement interface is its handling of service configura- 
tion data. We identified a set of meta-configuration, 
which is similar across a large set of services. For 
example, one critical pieces of meta-configuration 
captures whether a service is supposed to be running 
on a specific system. Common tools should be pro- 
vided for meta-configuration, while still retaining the 
flexibility for services to provide their own complex 
application-specific configuration. 

Meta-configuration must be available via a com- 
mon command set and API across all services. This 
allows rapid development of higher-level administra- 
tion tools which need no application knowledge to 
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perform common tasks. Service-specific configuration 
can also be stored in a format accessible by this API, 
but the choice to transition to the new configuration 
store is up to the application developer depending on 
their compatibility requirements. Through the com- 
mon commands, configuration rollback is provided. 
An administrator should be able to undo configuration 
or meta-configuration changes which rendered a ser- 
vice unstartable. 


A few constraints were also required for a practi- 
cal implementation in an operating system with strong 
compatibility requirements: 

¢ No application binary changes required for 
basic participation in the framework. 

¢ Compatibility for existing init.d services. The 
vast majority of services must continue to 
work, to account for lag time in new feature 
adoption by software developers. 


Extending UNIX Processes to Software Services 


SMF formalizes software “services’’, and intro- 
duces the tools to create, observe, manage and auto- 
matically restart them. A service is a long-lived soft- 
ware object (typically, a daemon) with a name, a clear 
error boundary, start and stop methods, and depen- 
dency relationships to other services on the system. 


A service may be composed of zero, one, or 
many processes. A comprehensive software service 
model must allow all of these process models, and 
take each into account when defining failures. We'll 
first consider a typical UNIX service which only has 
one process. These services are relatively straightfor- 
ward to manage and monitor. The single process 
makes process-to-service mapping easy to understand, 
as no complex lineage from parents to children needs 
to be tracked. If the process exits, the service is no 
longer being provided and must be restarted. Still, 
without a new mechanism to track process faults, most 
monitoring implementations would require services 
forgo the traditional reparent-to-init step. 


Even simple services can consist of multiple pro- 
cesses. For example, the Sendmail service usually 
includes two processes. Failure of either process 
should cause the service to be restarted. Traditional 
models for determining the relationship between these 
two processes fall down: they both re-parent them- 
selves to init, breaking monitoring implementations 
which rely on the parent-child relationship. Aside 
from the process name, it is difficult to tell they’re 
both part of the same service. 


Finally, we have a set of services which only 
appear transiently during the startup process. Usually, 
they exist to execute a command or small set of com- 
mands which change configuration state, such as 
informing the kernel about certain configuration 
parameters. It is important for SMF to understand a 
priori that a lack of long-running processes in these 
transient services does not constitute a fault. 
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In order to monitor the health for all three types 
of services, SMF introduces a kernel interface called a 
contract. A process contract allows a userland process 
to register an interest in a process and all of its chil- 
dren. The process receives reliable events when 
important changes occur, such as when all processes 
in the contract exit, a hardware error occurs in any of 
the processes, any processes in the contract coredump, 
or any process in the contract receives a fatal signal. 
All processes on the system must belong to a process 
contract, and all children of a process are part of the 
same contract until one creates a new contract. 


Service restarters are responsible for managing 
software services. A restarter can use process con- 
tracts to receive and respond to fault events for a ser- 
vice. Usually, a restarter will write a process contract 
for each new service it creates. Thus, events for each 
service will be sent individually to the restarter, where 
it can make decisions about how to respond to contract 
events and administrative requests. Each process can 
fail separately, but it is the restarter’s job to decide 
how a process failure should affect the service. 


Contract 4 


' Master Restarter 
(sve.startd) s 


| Contract 51 
sve:/network/inetd 








A 
Contract 2245 a 


_ svc:inetwork/smtp: ‘sendmail | 


| 





Figure 1: Contract model for software services. 


A restarter determines how to interact with a ser- 
vice through its meta-configuration, which is stored in 
the SMF repository. The SMF repository stores persis- 
tent configuration and meta-configuration values as well 
as shorter-lived status information about each service. 
Each service has a common set of meta-configuration: 

e The service name, in the form of a Fault Man- 
aged Resource Identifier (FMRI) is the unique 
identifier of the service on the system. The full 
FMRI for the sendmail service is: svc:/net- 
work/smtp: sendmail, but abbreviations like 
“sendmail” are available for interactive use. 

© Service dependencies specify the relationships 
between services. 
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¢ Method specifications tell the restarter how to 
invoke the services, and may include invoca- 
tions for start, stop, etc. 

¢ The restarter defined by the service is responsi- 
ble for starting, stopping, and restarting the ser- 
vice. Most services are managed by the default 
restarter, svc.startd. We’ve also implemented 
inetd as a service restarter. 

¢ Services can optionally include localized, 
human readable descriptions and documenta- 
tion references. 


A service may also use the SMF repository to 
store simple configuration information. This allows 
application and system service developers to avoid 
maintaining configuration file parsers for small, sim- 
ple configurations. 


We developed a comprehensive state model for 
managing services, which are always in one of the fol- 
lowing states: uninitialized, offline, online, degraded, 
disabled, and maintenance. These states have a consis- 
tent definition regardless of the service model. 


State Significance 


Uninitialized The initial state for all services. 
After evaluation by its restarter, the 
restarter will move the service to a 
maintenance, offline, or disabled 
state, 

The service has been disabled and 
is not running. 

The service is enabled, but is not 
yet running. Often, a service will 
remain offline due to an unsatisfied 
dependency. 

The service is enabled, has started 
successfully and is currently run- 
ning. 

The service is enabled, currently 
running but may be functioning at 
a limited capacity (e.g., reduced 
performance). The precise defini- 
tion of degraded is service-specific. 
The service is unavailable and can- 
not be automatically repaired by 
SMF. Administrative intervention 
is required to repair the service and 
clear the fault. 


















Disabled 


Offline 


Online 


Degraded 


Maintenance 


Table 1: State model for managing services. 


Services transition between these states either 
because of administrative action (system startup, service 
enable, service disable), or service error (core dump, 
starting too rapidly, hardware fault). The service’s abil- 
ity to transition to a given state is always additionally 
influenced by the states of its dependencies. 


To implement the service state model, SMF 
includes service restarters, which are themselves system 
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services (and hence subject to the state model). In our 
current work, a master restarter (svc.startd) and one del- 
egated restarter (inetd) have been developed. While 
each of these are concerned with the restart of UNIX 
processes, the restarter architecture also allows imple- 
mentation of non-process-based service models [6]. 
Each service restarter is responsible for defining appro- 
priate transitions between the available SMF states. 


Service Development 


The meta-configuration and other simple service 
configuration are delivered onto the system by a small 
XML file, known as a service manifest. When the sys- 
tem starts up, the individual manifests are imported 
into the main SMF repository. The configuration of 
the system, including administrative customizations 
and run-time service data is stored in the repository 
and accessed by a common set of commands and 
APIs. The repository also provides the transactional 
semantics required for recovery on failure as well as 
the previous configuration snapshots which allow for 
configuration rollback. 


In order to deliver an SMF-aware service, the 
administrator or software developer is required to cre- 
ate a service manifest. The manifest must include the 
meta-configuration for the service: a service name, the 
required methods, all dependency information, docu- 
mentation references, and any other configuration to 
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SMF currently provides two restarters: svc.startd, the 
master restarter, and inetd, the inet service restarter. Both 
restarters require service definition through a manifest. 


To provide a smooth upgrade path, service mani- 
fest creation can be an automatic step for inetd.conf ser- 
vices. We provide the inetconv utility to convert 
entries from inetd.conf to service manifests on operat- 
ing system upgrade and allow administrators to con- 
vert entries subsequently added to the file. This is 
made easy by the well-specified service model for 
inetd services, and the fact that their dependencies are 
well known: inetd itself must be started, and rpc ser- 
vices require rpcbind. 


Manifest creation for init.d scripts remains a man- 
ual process. The set of information required is well- 
defined, however two specific questions cannot be 
answered programmatically: 

1) What are this service’s dependencies? While 
we might be able to imagine a run-time checker 
or code analyzer for dependencies, these depen- 
dencies are often subtle and strongly tied to 
configuration. Automatic application depen- 
dency analysis would be a fascinating research 
topic of its own. 


The service author is usually best equipped to 
define the dependencies precisely. However, 
any consumer of the service can usually specify 
a dependency set that is sufficient for normal 


be stored in the repository. operation. 
sveadmclear OW 
— Uninitialized — Sees 
\ service disabled - 
can't read 
confi x be 
9 start service x 
sveadm Disabled 
disable aa 
ee, ya unresolvable ei a 
Maintenance 4 Bier Offline 
~~, J 
svcadm enable 
start failed 
unresolvable ~ service shutdown, 
error restart or disable 
service shutdown, . 
restart or disable 
dependencies partial failure 
Vv _ satisfied of service or_ >| Degraded 
"dependency om - 
Online - 
svcadm refreshA A dependencies satisfed 


and service is healthy 


Figure 2: svc.startd state transitions. 
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2) What is this service’s runtime behaviour? Does 
it have long-running processes that must be 
monitored, or is a lack of associated processes 
considered normal? 


The service manifest author is required to specify 
dependencies in the manifest. Dependencies define ser- 
vice startup order as well as the restart relationships 
between services. A dependency is said to be satisfied 
if all conditions of its definition are met. There can be 
multiple dependencies for each service and each depen- 
dency may declare interest in multiple services. Two 
attributes of a dependency aside from the actual ser- 
vices in the dependency must be defined. The grouping 
specifies how the state of the services specified in the 
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dependency are evaluated to determine whether a 
dependency is satisfied: 
° require_all: all named services are running 
(online or degraded) 
© require_any: at least one named service is run- 
ning (online or degraded) 
® optional_all: all named services must be either 
running (online or degraded), disabled, in main- 
tenance, or not present on the system 
© exclude_all: all named services must be either 
disabled, offline, in maintenance, or absent 


The restart_on property determines which events 
should cause the service to be stopped: 
© none: required only when the service is started 


<service_bundle type=’manifest’ name=’SUNWcsr:utmpd’> 


<service 
name='system/utmp’ 
type='’service’ 
version=’1’> 


<create_default_instance enabled=’true’ 


<single_instance/> 


<dependency 
name=’milestone’ 
grouping=’require_all’ 
restart_on=’none’ 
type=’service’> 


/> 


<service_fmri value=’svc:/milestone/sysconfig’ /> 


</dependency> 


<dependent 
name=’utmpd_multi-user’ 
grouping=’optional_all’ 
restart_on=’none’> 


<service_fmri value=’svc:/milestone/multi-user’ /> 


</dependent> 


<exec_method 
type=’method’ 
name=’start’ 
exec='/lib/svc/method/svc-utmpd’ 
timeout_seconds=’60’ /> 


<exec_method 
type=’method’ 
name='stop’ 
exec=’:kill’ 
timeout_seconds='60’ /> 


<stability value=’Unstable’ /> 


<template> 
<common_name> 


<loctext xml:lang=’C’> utmpx monitoring 


</loctext> 
</common_name> 
<documentation> 


<manpage title=’utmpd’ section='1M’ manpath=’/usr/share/man’ /> 
<manpage title=’utmpx’ section=’4’ manpath="/usr/share/man’ /> 


</documentation> 
</template> 
</service> 
</service_bundle> 


Figure 3: utmpd manifest. 
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° error: stop the service if the dependency fails due 
to hardware, software or administrative error 

restart: stop the service if the dependency stops 
for any reason 

e refresh: stop the service if the dependency stops 
or is refreshed 


Explicit dependency specification makes parallel 
service startup easy, regardless of whether the system is 
booting for the first time, or if a set of services had failed 
and are being restarted. Services are always started as 
soon as their dependencies are satisfied, so maximum 
parallelism for starting services is always achieved. 


Unifying application deployment also provides a 
single location for access to advanced features. A ser- 
vice can be run with limited Privileges [11], or as an 
unprivileged or non-root user. Service management 
privileges may be given to authorized users without 
giving them full root access. A service may be bound 
to specific resource management limit and goal sets. 
All of this can be specified as service configuration 
with no code changes to the application itself (see Fig- 
ure 4). Management of Solaris Zones [9] is simpler as 
SMF is available to each zone administrator. 


Service Administration 


The primary benefit a unified service manage- 
ment framework brings to system and service adminis- 
tration is a meaningful system-level view of all critical 
applications (see Figure 5). Services in an unexpected 
state are sorted at the bottom of the output for simple 
human consumption. 
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A primary goal of the SMF administrative model 
is to make common questions about services easy to 
answer and make common system administration 
tasks simple to perform. To evaluate the simplicity of 
the SMF model, the following provides examples of a 
few of these questions that were particularly difficult 
to answer in the init.d and inetd models prior to SMF. 


What processes make up this service? 
$ svcs -p sendmail 


STATE STIME FMRI 

online May_06 svc:/network/smtp:sendmail 
May_06 9456 sendmail 
May_06 9458 sendmail 


What's wrong with my system? (This will be covered 
in more detail in the next section.) See Figure 6. 


What services does my service require in order to 
start? See Figure 7. 


Which services wont be able to run if I disable this 
service? See Figure 8. 


What services are available on this system? See Fig- 
ure 9. 


Messaging is also under the control of the 
administrator rather than the service. Messages previ- 
ously emitted to console are stored in a per-service 
logfile, where they can be perused as part of post- 
mortem debugging or other diagnosis. They won’t be 
lost to an insufficient terminal buffer size. 


SMF defines a specific set of common adminis- 
trative actions which can be applied to services: 


## svecfg -s <svce> setprop start/user = astring: daemon 
## svecfg -s <svc> setprop start/group = astring: daemon 
# svcadm refresh <svc> 
## svcadm restart <svc> 
Figure 4: Configuring a service to run as “daemon” user and group. 
$ svecs 
STATE STIME FMRI 
[nal 
legacy_run May_02 Irc:/etc/rce3_d/S81volmgt 
legacy_run May_02. = Ire: /etc/rc3_d/S84appserv 
legacy_run May_02 lre:/etc/rce3_d/S90samba 
online May_02 svc:/system/svc/restarter:default 
online May_02 svc:/network/pfil:default 
online May_02 svc:/network/loopback:default 
online May_02 svc:/system/filesystem/root:default 
online May_02 svc:/system/filesystem/usr:default 
Pes 
Figure 5: Abbreviated svcs output, including legacy services. 

$ sves -x 
svc:/network/smtp:sendmail (sendmail SMTP mail transfer agent) 
State: maintenance since Tue May 10 18:35:41 2005 
Reason: Method failed repeatedly. 

See: http://sun.com/msg/SMF-8000-8Q 

See: sendmail (1M) 

See: /var/svc/log/network-smtp:sendmail.log 
Impact: This service is not running. 


Figure 6: Checking the sendmail service. 
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¢ enable — Mark the service as enabled and start 
the service after all dependencies are satisfied. 

¢ disable — Mark the service as disabled, stop the 
service, and do not allow it to start again. 

e refresh — Reload service configuration and run 
the service’s refresh method (if a_ refresh 
method is defined by the service). 

° restart — Stop the service, then start it after its 
dependencies are satisfied. 

¢ clear — Mark a service in the maintenance state 
as repaired, and if it is enabled allow it to start 
after all its dependencies are satisfied. 


Administrators use svcadm to perform these 
administrative actions. Administrative intent is always 


$ svcs -d sendmail 
STATE STIME FMRI 


online Aug_12 svc:/system/identity:domain 
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preserved; disabling a service is guaranteed to persist 
across even patch and upgrade boundaries, where that 
was difficult in the past. The separation of administrative 
action and service state allows easier evaluation when 
system state doesn’t match the administrative desire. 


The use of a single API to take administrative 
action and change service configuration allows one 
point for security enforcement. Thus, SMF interoper- 
ates intimately with Role Based Access Control [11] 
mechanisms. It is easy to give a user just the ability to 
take action on a service (e.g., restart the service), but 
not change the service’s configuration (see Figure 10). 
An administrator may also delegate authorizations with 
more granularity, giving privilege for an application 


online Aug _12 svc:/system/filesystem/local:default 


online Aug_12 svc:/network/service:default 


online Aug_12 svc:/milestone/name-services:default 
online Aug_12 svc:/system/filesystem/autofs:default 


online Aug_12 svc:/system/system-log:default 


Figure 7: Determining services needed by sendmail. 


$ svcs -D system-log 


svce:/system/auditd:default 
Aug 12 svc:/application/print/server:default 


Aug_12 svc:/milestone/multi-user:default 


STATE STIME FMRI 

disabled Aug_12 

disabled 

disabled Aug_12  sve:/network/rarp:default 
online 

online 


5:55:34 svc:/network/smtp:sendmail 


Figure 8: Determining services which require system-log’s. 


$ svcs -a 
fesse 

disabled 
disabled 
disabled 
disabled 
disabled 
disabled 
easseeoih 


Aug_12 svc:/network/iscsi_initiator:default 
Aug_12 svc:/system/metainit:default 

Aug_12 sve:/network/ipfilter:default 

Aug_12 svc:/network/rpc/nisplus:default 
Aug_12 svce:/network/nis/server:default 

Aug 12  sve:/network/ldap/client:default 


Figure 9: Available service listing. 


$ id 
uid=37436(lianep) gid=10(staff) 
$ grep lianep /etc/user_attr 


lianep::::auths=solaris.smf.manage 
$ sves sendmail 

STATE STIME FMRI 
disabled 


$ svcadm enable sendmail 

$ sves sendmail 

STATE STIME FMRI 
online 
S$ svcadm refresh sendmail 

S$ svcs sendmail 

STATE STIME FMRI 
online 
$ svecfg -s sendmail delpg autofs 
svecfg: Permission denied. 


18:51:56 svc:/network/smtp:sendmail 


18:52:43 svce:/network/smtp:sendmail 


18:52:55 svce:/network/smtp:sendmail 


Figure 10: Delegating service management authorizations. 
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administrator to only manage and change the configura- 
tion of a single service [10]. 


Finally, system security can be easily configured 
by creating a profile, which explicitly specifies which 
services are enabled and disabled on a system. To sat- 
isfy the common system configuration goal of no 
unencrypted network login services running, we pro- 
vide the limited networking profile. Applying this pro- 
file explicitly disables all unencrypted network login 
services and enables other important services like ssh 
(see Figure 11). 


Service Diagnosis and Self-Healing 


svcs -x is a quantum leap forward in diagnosing 
problems with systems. It provides a very powerful par- 
adigm. It describes what’s going on in plain language 
with simplified output. It includes documentation refer- 
ences, as direct access to more information speeds the 
repair process. It points to an online knowledgebase; 
the website link included in the output contains the 
most up-to-date information on how to resolve the 


d# sves telnet 
STATE STIME FMRI 
online 


ite cd /var/sve/profile/ 
## cat generic_limited_net.xml 
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specific problem seen. Finally, svcs -x gives an assess- 
ment of the impact of each problems. If the specific 
issue effects no other services, that is stated explicitly. 
If other services are affected, svcs -xv will list them. 


As the system (through dependency information) 
can determine the root cause of the problem, it can 
point the administrator directly to the component that 
must be repaired (see Figure 12). 


In many cases, though, the administrator never 
needs to handle an error manually. Failure of a service 
due to hardware error, software bug, or administrative 
error can usually be resolved by restarting the service. 
This is handled automatically, with no administrative 
intervention. SMF also logs the error cause, to the 
extent it is known at the time of failure. This automatic 
recovery significantly reduces administrative costs. 


To implement service restartability, SMF needed a 
way to detect when an error occurred in the service. 
Contracts provide a generic mechanism to express a 
relationship between a process and the kernel-managed 
resources it depends upon. The process contract allows 


17:49:15 sve:/network/telnet:default 


<service_bundle type=’profile’ name=’generic_limited_net’ 
xmlns:xi=’http://www.w3.org/2003/XInclude’ > 


Lieto 4 


<service name=’network/ssh’ version='1’ type='’service’> 


<instance name=’default’ enabled=’true’/> 


</service> 


[oo =d 


<service name=’network/telnet’ version=’1’ 


type=’service’> 


<instance name=’default’ enabled=’false’/> 


</service> 


a 


</service_bundle> 


## svecfg apply generic_limited_net.xml 
# sves telnet 

STATE STIME FMRI 

disabled 


17:49:40 svc:/network/telnet:default 


Figure 11: Viewing and applying the limited network profile. 


S$ svcs -xXv 


sve:/system/filesystem/local:default (Local filesystem mounts) 
State: maintenance since Tue Sep 27 19:03:43 2005 
Reason: Start method exited with S$SMF_EXIT_ERR_FATAL. 


See: http://sun.com/msg/SMF-8000-KS 


See: /var/svc/log/system-filesystem-local:default.log 
Impact: 23 dependent services are not running: 


svc:/system/sysidtool:net 
svce:/network/rpc/bind:default 
svce:/network/nfs/status:default 


svc:/network/nfs/nlockmgr:default 


svce:/network/nfs/client:default 


svc:/system/filesystem/autofs:default 


bed 


Figure 12: Service diagnosis of filesystem mount failure. 
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development of sophisticated restarters, which create a 
fault boundary around a set of processes, and receive 
and respond to events on processes within that boundary. 


Implicit in service recovery is that the framework 
itself must also be fully restartable in the face of fail- 
ures. An error of any user-land framework component 
all the way back to init can be caught, and the frame- 
work component itself will be restarted (see Figure 
13). A transactional repository is required in order to 
implement the algorithms which recover from failure 
at any point. 


Systems with sophisticated hardware error han- 
dling [7] make software recovery after error even 
more critical. Prior to SMF, the operating system 
developer when handling a hardware failure was given 
only a few unenviable options: kill the affected 
process and risk cascading failure, or restart the entire 
system. The operating system could not determine the 
broader effect of a faulted cell on a DIMM without 
inter-service dependency information. SMF manages 
error flow between services so that failures can be 
handled gracefully, by shutting down only affected 
processes and services which depend upon them. 


Availability 


The first version of the Service Management 
Facility is an integrated component of the Solaris 10 
Operating System, released in January 2005. Themost 
recent copy of Solaris may be obtained free of charge 
at http://www.sun.com/software/solaris/ . 


The Service Management Facility is also an inte- 
gral part of OpenSolaris, which may be downloaded as 
source or binaries at http://opensolaris.org/os/. The 
OpenSolaris SMF community contains information 
and discussion about service development and man- 
agement in Solaris and OpenSolaris: http://opensolaris. 
org/os/community/smf/. 


| fee & | 
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Experience 


The first proof of concept and vetting of the 
design occurred when transitioning approximately 100 
system services from their init.d script components into 
SMF. We wanted to confirm certain aspects of the 
SMF design, as well as take advantage of SMF bene- 
fits for Solaris service administration. The majority of 
services delivered as part of the Solaris operating sys- 
tem were migrated to SMF as part of its initial integra- 
tion. We'll explore this case study here. 


A small engineering team was responsible for 
creating nearly all of the 100 manifests for system ser- 
vices. A major lesson was that once a developer gains 
familiarity with the SMF model, manifest creation 
tends to take no more than a few hours per service, 
often including necessary testing. Experience with one 
manifest is directly leveraged for subsequent manifest 
creation. The fundamentals for basic service coopera- 
tion in SMF are relatively easy to grasp and do not 
seem to require significant training. The more sophis- 
ticated aspects of the service model tend to not be 
explored by the average service author, which we 
interpreted as success in our goal to make simple ser- 
vices easy to define. 


As expected, the most challenging part of mani- 
fest creation was researching proper dependencies for 
each service. Sometimes the initial dependency analy- 
sis was incomplete — but, both point fixes for signifi- 
cant problems and longer term dependency additions 
to fill gaps were easy to perform. 


We learned that someone familiar with the ser- 
vice implementation will be able to write a signifi- 
cantly better manifest than someone merely conversant 
in the service. A service manifest written by an end- 
user of the software is sufficient for that user’s config- 
uration, but not all potential uses and configuration of 
the service. Encouraging service authors to write 





Figure 13: SMF framework restart relationships. 
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manifests for their software has significant value, even 
when a basic manifest has already been written. 


Console interaction was more of a challenge than 
anticipated. When system startup was serial, console 
ownership was easy as it simply passed linearly from 
one script to another. SMF’s parallel startup required 
tracking the console owner at all times, in case a sys- 
tem maintenance mode (also known as sulogin) was 
required to repair the system. The console ownership 
problem is a compelling reason for Solaris to provide 
access to the standard console login prompt as early in 
the boot sequence as possible. This convention is par- 
ticularly helpful in a repair scenario because as much 
of the system as possible is available for use during 
the repair, rendering the repair environment less 
restrictive and more familiar. 


Confirming the SMF design with critical system 
services started early in the boot sequence was likely a 
bit painful for the very early adopters. However, the 
lessons learned through actual service debugging 
informed significant usability enhancements which 
may not have been given the attention they deserved 
had SMF been an optional feature used only for a 
small subset of services. 


Conclusions 


A full-featured application and system service 
management infrastructure must reduce management 
complexity, increase application availability, and save 
administrators time. In order to provide all of these 
capabilities for a wide range of application models, the 
service must be elevated to a first-class administrative 
object which can be observed and managed. The Ser- 
vice Management Facility provides all of these bene- 
fits while requiring no changes to application binaries. 


SMF reduces complexity by unifying and simpli- 
fying common administrative tasks across a broad set 
of applications. It increases application availability by 
detecting many critical errors and recovering from 
them automatically. Finally, it saves administrators 
time; when automatic recovery from failures is impos- 
sible, a complete management interface guides the 
administrator to the faulty component. 
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for Securing Legacy Resources 
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ABSTRACT 


Users of a network system often require access to legacy resources. Providing this access is a 
difficult task for system administrators because the access protocols for those resources are typically 
insecure. A common approach is to develop a custom wrapper or proxy that securely processes user 
requests before forwarding them to the legacy server. The problem with this approach is that 
administrators must develop a custom solution for every resource. We believe that there are common 
requirements for managing these resources that can be addressed from a more centralized model. 
The userspace queuing extensions of the Netfilter firewall modules provide a generic environment in 
which protocol-aware deep packet filters can be constructed to enhance the security of resource 
access protocols. We employ this environment to strengthen two commonly used legacy protocols, 
and compare their requirements. We show that it is possible to secure legacy resources with minimal 
degradation in performance. We also discuss considerations for development of a deep packet filter 
toolkit to aid system administrators in securely managing legacy network resources. 


Introduction 


It has been our experience that, for a variety of 
reasons, production networks typically house several 
legacy resources. It may be the case that an applica- 
tion critical to the users of a network is no longer 
under active development, and no adequate substitute 
has been found. If a newer solution is available, other 
factors such as deployment cost or compatibility con- 
cerns may prevent its installation. The result is that 
system administrators must provide access to 
resources that employ inherently insecure protocols, a 
concept in conflict with their responsibility to main- 
tain the security of the network and its resources. 


By developing a way to manage these resources 
securely, administrators can continue to provide the 
needed services without endangering other compo- 
nents of the system or risking the integrity of the 
resources themselves. Currently, the only way to 
achieve this is to develop a custom environment in 
which to “‘wrap” each resource. No toolkit or frame- 
work exists that allows administrators to secure exist- 
ing protocols with minimal effort. We believe that 
despite the arbitrary complexity of a protocol, certain 
common requirements exist for administrators that 
allow the development of such a toolkit. 


As the basis for this toolkit, we employ the 
userspace queuing extensions of the Netfilter firewall 
[17] distributed with the Linux operating system. The 
processing capabilities provided by the userspace 
extensions allow us to collect any information deemed 
appropriate for a resource request before determining if 
the traffic should be allowed to contact the server. We can 
delay, alter, or reroute packets, interactively challenge 
authenticity, collect system status information, or anything 
else dictated by the protocol and administrative require- 
ments of the resource at hand. 


To illustrate both the power and flexibility of this 
approach, we consider two common protocols and the 
weaknesses they contain. In both cases, newer versions 
or extensions exist that address those problems, but we 
assume that some other consideration such as cost or 
compatibility prevents them from being deployed. This 
provides an example of what we believe to be a com- 
mon issue in typical network system environments. 
Specifically, we discuss the Ipr line printer protocol as 
defined by RFC 1179 [9] and the Network File System 
(NFS) protocol, version 3 [2]. Both of these systems 
provide essential resources to users, yet neither pro- 
vides any strong authentication of those users. 


Using the Netfilter userspace queuing extensions, 
we develop strong user authentication for each of these 
protocols. In the case of Ipr, we employ public-key 
cryptography to ensure that users can only submit print 
jobs under their own identities. This prevents users 
from constructing raw print job control files in order to 
print under a false identity, possibly bypassing quota 
enforcement. Our technique also directly authenticates 
users, allowing administrators to open the print server 
to outside domains, such as wireless networks, provid- 
ing users the ability to print securely from their lap- 
tops. For the NFS example, we use symmetric-key 
challenge-response authentication to secure filesystem 
mount requests. In order to mount a secured filesys- 
tem, a client must be in possession of the current key 
and know the format of the challenge issued by the 
firewall. This prevents unauthorized client machines 
from mounting remote filesystems simply by pretend- 
ing to be authorized hosts. 


Both of our extensions significantly improve the 
security of the original protocols, while introducing mini- 
mal overhead. They demonstrate how two vastly differ- 
ent protocols can be secured by the same basic technique, 
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and suggest a set of tools that could be used by adminis- 
trators to secure legacy applications and protocols with 
significantly less effort than is currently required. 


Design Goals 


Faced with the problems of securing and manag- 
ing legacy resources in a network, we define the fol- 
lowing requirements for a useful solution: 

¢ Neither client nor server components of any 
application should require modification to take 
advantage of the features we provide. Many 
existing approaches, such as application prox- 
ies, are not always feasible since they require 
application code to be rewritten or expanded. 

¢ The solution must be generic enough to support 
any application with minimal overhead. Each 
application will impose different requirements 
on the solution. It should allow administrators to 
develop complex management systems without 
developing entire filtering applications from 
scratch. 

¢ The system must be able to adapt to changing 
conditions in the environment, adjusting its filter- 
ing behavior accordingly. For example, depend- 
ing upon application requirements, traffic from 
authenticated clients might not need to be filtered 
for a given period of time. 


General Related Work 


Because we deal with protocol-specific traffic 
filtering, the most relevant existing work relates to 
deep packet filtering [7]. Several authors have 
acknowledged that the basic address/protocol-based 
filtering provided by traditional firewalls is not suffi- 
cient for many applications. In cases where filtering 
requirements are tied directly to a given protocol, 
stock firewalls are not appropriate. Instead, a filter that 
examines protocol-specific information contained in 
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Because deep packet filtering incurs a higher cost than 
traditional header-based filtering, much of the research 
attention has been paid to improving performance of 
these filters, even via hardware solutions [4, 6, 18]. 
Very recently, it has been suggested [5] that forthcom- 
ing firewalls must include some level of deep packet 
filtering capability, due to the increasing number of 
application-level attacks. We agree, and the toolkit we 
develop is an essential step toward integrating these 
protocol-aware filtering capabilities in commonly 
existing software-based firewalls. 


Implementations 


Here we describe two prototypical applications 
chosen to demonstrate the capabilities of our tech- 
nique. Each is exemplary of different components of 
the system interacting with different protocols. Both of 
them were implemented on a six machine testbed in a 
secure laboratory [12]. Figure | illustrates the testbed 
layout. The firewall separates two private subnets. On 
one subnet resides three test clients that interact with a 
server residing on the other subnet. They are con- 
nected by a path that travels through the firewall. This 
is designed to emulate a typical setup in which a 
perimeter firewall protects a server or set of servers 
from client traffic, whether it originates from an inter- 
nal network or an outside internet. We also place a 
client-class machine on the server subnet to provide 
control data for performance analysis. Since it is on 
the same subnet, the control client need not traverse 
the firewall to interact with the server. The machine is 
composed of the same hardware that constitutes the 
other three clients. By running identical client-server 
applications on the control client and the firewalled 
client, we can derive the overhead imposed by our 
toolkit. Note that a second path exists through which 
traffic could possibly travel from clients to servers, but 
this connection is not a part of the testbed. It is used 


the packet payloads provides the required service. 


for administrative purposes in the lab environment. 








control 
client 
server 










experimental 
client 

experimental 
client 

experimental 
client 





Figure 1: Prototype testbed. 
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iptables has a modular extension interface that 
allows developers to introduce additional features. 
One module included in the distribution introduces a 
QUEUE target that stores packets in a kernel data struc- 
ture until they are retrieved by a userspace daemon. 
This daemon can then analyze the packet and its con- 
tents, and determine an appropriate course of action 
using the resources provided to userspace programs. 
Figure 2 illustrates the components of a firewall sys- 
tem that uses this approach. Since it allows us to 
implement more detailed packet analysis than the 
built-in features of iptables, we employ this module in 
the implementation of our deep packet filter daemons. 


NFS Prototype Implementation 


The Network File System [2] allows filesystems 
on a sever to be exported over a network to multiple 
clients. Clients can mount the filesystem, which then 
appears as part of the local directory structure; net- 
work operations are transparent to the user as he inter- 
acts with the remote files. 


Since the filesystems being exported to remote 
clients may contain sensitive information, we must 
ensure that only authorized clients are able to mount 
them. The stock implementations of NFS provide sim- 
ple host-based authentication by comparing the IP 
address of a client from which a mount request origi- 
nates against a table of pre-authorized client addresses. 


server 








4. Application receives packet. 


3. Daemon processes packet; sends to application. 


192.168.2.5 


1, Client generates pakcet and sends to server. 


Towards a Deep-Packet-Filter Toolkit for Securing Legacy Resources 


To further increase security, some environments employ 
client MAC address filtering at the network switch level. 


Even in this case, however, a malicious user can 
gain unauthorized access to the filesystem. He needs 
only to set manually the MAC address for his network 
interface, configure his client with an authorized IP 
address, and unplug a legitimate system. He can then 
connect to the switch, replacing the connection of the 
system whose MAC and IP addresses are being 
spoofed. By taking these steps, the user presents a 
client to the system that, in most configurations of the 
currently prevalent protocols, is indistinguishable 
from a valid, authorized client. These considerations 
dictate the need for stronger authentication of clients 
wishing to mount remote filesystems. 


Related Work 


Despite the presence of new, more secure, file 
sharing paradigms [13, 3], traditional NFS is still 
widely used. Accordingly, some attempts have been 
made to strengthen authentication in the NFS protocol 
itself. Ashley, et al. [1] introduced role based access 
control to the protocol. O’Shanahan [15] replaced the 
UID-based authentication in NFS with a public-key 
cryptosystem. A less intrusive modification was pro- 
posed by Goh, et al. [8]. In their system, the client 
intercepts all filesystem operations and encrypts them 
on the fly so that they are secure when stored on the 
server. The underlying NFS structure isn’t changed, 


-p all -s 192.168.2.5 -j QUEUE 


2. Packet matches queue target; sends to userspace daemon. 


Figure 2: Packet filtering with userspace queues. 


IPTABLES -A FORWARD TCP -s SCLNTNET --dport sunrpc 
IPTABLES -A FORWARD TCP -s S$CLNTNET --dport nfs 
IPTABLES -A FORWARD UDP -s SCLNTNET 


p 
Pp 
“Pp 
IPTABLES -A FORWARD -p UDP -s $SERVER 
Pp 
P 
Pp 
“Pp 


IPTABLES -A FORWARD TCP -s SSERVER 
IPTABLES -A FORWARD TCP -s S$CLNTNET --dport sunrpc 
IPTABLES -A FORWARD TCP -s SCLNTNET --dport sunrpc 
IPTABLES -A FORWARD TCP -s SCLNTNET --dport sunrpc 
IPTABLES -A FORWARD -p UDP -s SCLNTNET --dport sunrpc 


Figure 3: Initial ruleset for authenticated NFS. 
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--tep-flags syn syn -j ACCEPT 
--syn -j ACCEPT 
--dport nfs -j ACCEPT 
--sport nfs -j ACCEPT 
--tep-flags ack ack -j QUEUE 
--tep-flags ack ack -j QUEUE 
--tep-flags urg urg -j QUEUE 
--tcp-flags fin fin -j ACCEPT 
-j QUEUE 
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but the data being stored within that structure are mod- 
ified transparently to the user. Our approach is differ- 
ent in that no components of the system require modi- 
fication of any kind. Server and client daemon behay- 
ior is unchanged, as are the data and metadata stored 
in the remote filesystem. The only component we 
introduce is a client-side authentication daemon that 
responds to challenges issued from the firewall during 
mount attempts. Assuming that clients are not able to 
bypass the mount process, we gain a more secure NFS 
environment. If, however, a client successfully 
guesses a remote filehandle, it can modify the associ- 
ated file by manually constructing NFS packets and 
interacting directly with the server daemon. Our 
approach, like any system that does not authenticate 
every client-server interaction, cannot prevent this 
form of compromise. 


Connection Life Cycle 


In order to challenge traffic destined for NFS 
mount daemons, we must first learn the port at which 
those daemons are running. Since the portmapper 
query for this information will always be destined for 
the sunrpc port of the server, we can easily intercept 
this traffic, remember that a client has requested this 
port information, then allow the request to proceed. 


Dynamic Rules 


Our obligation to intercept any portmapper 
request defines the initial iptables ruleset, illustrated in 
Figure 3. 


When the server responds, we also intercept that 
information before forwarding it to the client, and 
extract from the reply the ports on which client’s mount 
request will be made. We then inject a new rule into the 
firewall that QUEUEs the client’s forthcoming mount 
request to the authentication daemon. The system can 
handle requests from multiple clients simultaneously, 


NFS Server Firewall 


isting request 


il 13 port 





NFS Client NFS Server Firewall 
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provided that the Netlink queue buffers do not overflow. 
If this occurs, packets are dropped as though the firewall 
discarded them, or network congestion occurred. 


Authentication Mechanism 


We include an encrypted challenge-response 
scheme to authenticate remote clients. When a client 
issues a mount request, it is intercepted by the 
userspace daemon on the firewall and queued in a data 
structure to be either accepted or dropped depending 
on the result of the authentication process. The filter- 
ing daemon maintains a state variable for each packet 
that records that packet’s sequential arrival order at the 
firewall. Since the value of this variable is effectively 
unique for each packet over the lifetime of the system, 
we can use it to key a data structure that records all 
unanswered challenges. Our prototype uses a linked 
list to store this information, allowing multiple simul- 
taneous client connections. Figure 4 illustrates the 
structure of a single list node. 


struct list_element 

{ 
ipq_packet_msg_t *packet; 
long packet_id:; 
long nonce; 
struct list_element ‘prev; 
struct list_element *next; 


Figure 4: NFS packet queuing structure. 


The userspace daemon issues a challenge to the 
client consisting of a symmetrically encrypted id- 
nonce pair, and waits in the background for additional 
traffic to arrive. This traffic may be another mount 
request that triggers the same process, lengthening the 
queue, or it may be a client’s response, determining 
the fate of the mount request. The daemon can apply 
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Figure 5: Authenticated NFS service. 
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to a packet the same primitive filtering targets as the 
kernel firewall. A library interface provided by the 
queueing extension module described earlier provides 
these targets, allowing the daemon to ACCEPT or 
DROP a packet. If the client’s response is satisfactory, 
an ACCEPT verdict is issued on the original packet, 
and the mount request is processed by the server. If 
not, then the packet is dropped, and the server never 
sees the request. With encryption keys pre-loaded on 
systems and restricted to root access, a malicious user 
must successfully root-compromise a legitimate client 
to subvert the protocol. Spoofing network addresses of 
any type is no longer sufficient to gain access to the 
remote filesystem. 


Assuming that authentication succeeds, the fire- 
wall simply forwards all other NFS traffic between 
these two systems; no further intervention is imposed 
until the session ends. Since unmount requests are also 
processed by mountd, an authentication challenge will 
be issued at this stage as well. This is a desirable 
behavior, as it prevents an unauthorized user from 
unmounting a filesystem from beneath a legitimate 
user. Figure 5 illustrates the message passing in the 
authenticated system. 


Host Failures 


In the event that the server crashes, the client 
loses connectivity with the remote file system, but 
only while the server is down. Since authentication 
only happens during mounts, a reboot of the server 
simply delays NFS calls until the reboot is complete. 
The client does not have to remount the filesystem 
unless something catastrophic enough to change the 
filehandle on the server side has occurred. 


Should the client crash, all mount information is 
lost during the reboot. the client will need to remount 
the remote filesystem, requiring that the authentication 
process be repeated. The firewall and server have no 
way of knowing that the client crashed, and assuming 
that the mount request is legitimate would open the 
system to attack. 


Performance 


To measure the performance overhead associated 
with the system, we run benchmarks from two client 
machines and determine achieved bandwidth in the 
filesystem. One client mounts the NFS share through 
the firewall, while the control client resides on the 
same subnet and has a locally switched connection to 
the server. Figure | illustrates the distinction. 


Measurements were taken using the [OZone 
Filesystem benchmark suite [10]. First, a 32 MB file is 
written to the NFS share. The suite records the amount 
of time required to write the file and calculates the 
achieved bandwidth. A similar approach is used to 
measure reading performance. We present benchmark 
results for sequential and random writing and reading 
of records ranging from 2KB to 1024KB in a randomly 
generated 32 MB file. Data are presented within 95% 
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confidence intervals generated from 30 independent 
measurements of the system’s performance. 


The benchmark suite allows us to compensate for 
many caching effects. We employ some of these fea- 
tures to present clearer results. Between every test, the 
filesystem is unmounted; this clears the filesystem 
buffer cache, and forces the next test to interact 
directly with the filesystem instead of a faster cached 
copy. We also employ a setting that flushes the proces- 
sor caches between each test to ensure that no data are 
cached locally during read tests. 


It is important to note that the benchmarks mea- 
sure achieved bandwidth in the filesystem only after it 
is mounted. Time required to mount and unmount the 
filesystem is not included the results. This means that 
the authentication mechanisms used must be timed 
separately. What we measure here is how much the 
presence of the firewall between the client and server 
slows down normal operations. 


First, we consider the slowdown imposed by the 
firewall on sequential read and write operations. Fig- 
ures 6 and 7 illustrate the performance difference 
between the firewalled and control clients. In both 
cases, the slowdown is likely to be less than 1% for 
any record size. We consider this a negligible amount 
of overhead. 


Next, we examine the random read and write 
cases. Figure 8 illustrates another negligible slow- 
down. In this case, the worst slowdown is just under 
3%, with most cases falling well below 1%. The only 
case in which we see a significant slowdown imposed 
by the firewall is randomly reading records from the 
file, as illustrated in Figure 9. Here, the extreme case 
imposes a slowdown of about 11%, with most cases 
incurring a slowdown near 6%. Given that only one 
type of operation incurs a significant slowdown, and 
that slowdown, on average, is only about 6%, we 
argue that the overhead imposed by the system is 
small enough to justify the additional security afforded 
by our approach. 


The time required to mount the remote filesys- 
tem, including the encrypted authentication exchange, 
was 268+11 uS on the firewalled client. The control 
client, which has a direct connection to the server and 
does not perform an authentication exchange, mounted 
the filesystem in 16+0.2 US. Relative to the perfor- 
mance of the control client, the firewall introduces sig- 
nificant overhead in mount the remote filesystem. 
Note, however, that this overhead is encountered only 
when the filesystem is mounted, and the real time 
required to mount the directory is still extremely 
small. Amortized over the lifetime of the directory 
mount, the overhead is insignificant. 


LPR Prototype Implementation 


RFC 1179 [9] defines the widely used Ipr printing 
protocol. It contains no support for secure authentication 
of users. In environments where printing is monitored 
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record size 
Figure 6: Firewall performance relative to control 
client. Sequential read of 32MB file. 95% confi- 
dence intervals. 


percent change 





1 10 100 1000 
record size 
Figure 7: Firewall performance relative to control 
client. Sequential write of 32MB file. 95% confi- 
dence intervals. 


monitored or under quota enforcement, this issue is 
significant. It means that a user can easily submit 
printing jobs under another user’s ID, thereby bypass- 
ing the cost associated with his job request. This same 
issue complicates the matter of allowing authorized 
users to print without restriction based on network 
topology. Ideally, an associate of a department should 
be able to print to that department’s printers whether 
using his desktop workstation or his laptop from 
home. Since, however, the protocol doesn’t support 
strong authentication of users, allowing such open 
access would certainly result in numerous unautho- 
rized printing requests. 


Related Work 


Many approaches exist to solve this problem. 
LPRng [16] introduced a new implementation of the pro- 
tocol that supports the RFC 1179 interface while provid- 
ing additional functionality such as Kerberos authentica- 
tion [14] and public-key cryptography support. Most 


Deverick and Kearns 


percent change 
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record size 
Figure 8: Firewall performance relative to control 
client. Random read of 32MB file. 95% confi- 
dence intervals. 


percent change 





l 10 100 1000 
record size 
Figure 9: Firewall performance relative to control 
client. Random write of 32MB file. 95% confi- 
dence intervals. 


secure printing protocols that conform to the Ipr speci- 
fications are extensions of LPRng. While this 
approach does solve the authentication problems we 
outline, LPRng is only compatible with UNIX-like 
operating environments. Windows clients, while capa- 
ble of printing to RFC 1179 Ipr servers, do not under- 
stand the extensions in LPRng. Accordingly, they can- 
not take advantage of the authentication attributes pro- 
vided by the implementation. We offer an enhance- 
ment to the standard RFC 1179 protocol that requires 
no changes on either the client or server side, yet still 
introduces a strong authentication scheme. 


A common approach to strengthening the secu- 
rity of a legacy protocol is to tunnel all of its traffic 
through an encrypted channel, usually with SSH. This 
does provide stronger security, but the degree of 
improvement is highly application specific. In this 
case, it only partially authenticates the user. An SSH 
tunnel authenticates the connection, but not the use of 
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the protocol at hand. Suppose a legacy line printer sys- 
tem is protected by an SSH tunnel, through which end 
users must forward jobs. 


The result is that only authorized users will be 
able to submit jobs, but there is no enforcement that 
those jobs be submitted in the correct name. A user 
may have an account on the system, allowing her to 
establish an SSH tunnel to the print server, but nothing 
stops her from submitting the job under false identifi- 
cation (as described below) once that tunnel is in 
place. A similar property holds true for any encryption 
or tunneling scheme, including virtual private net- 
works or [IPSec tunnels. If the underlying protocol 
doesn’t directly support user-specific strong authenti- 
cation, then secure channels do not provide robust 
authentication in typical production environments. 


RFC 1179 requires that clients submit a print job 
control file to the server in order to process a new job. 
Contained in this control file is the username of the 
entity submitting the job. The problem is that this field 
in the control file is presented in cleartext. Nothing 
prevents a malicious user from altering the contents of 
this file before the job is submitted, thereby tricking 
the server into printing the job under someone else’s 
userid. The scheme that we offer forces the user to 
demonstrate with very high probability that he or she 
is, in fact, the user designated in the control file for the 
job being submitted. 
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Simply put, we intercept the TCP syn packet 
associated with the connection to Ipd, the print server 
process, and delay it at the firewall until we receive a 
digitally signed message from the client informing us 
of the owner of the incoming job. Once this informa- 
tion is stored, the TCP connection is allowed to pro- 
ceed, and when the control file begins to arrive at the 
firewall, the daemon extracts the job user information 
from the file and compares it to the signed username 
previously received. If they match, then we know that 
the user for this job is authentic, so we allow all other 
traffic on this connection through. If not, then we 
immediately tear down the connection. The server will 
abort the job, and the client’s only option is to restart 
the job submission process. 


User Authentication in Advance 


The initial ruleset for our approach is as illus- 
trated in Figure 10. 


This small ruleset simply propagates all Ipd- 
bound packets to our userspace daemon for process- 
ing, including the TCP packets used to construct the 
connection over which the job transfer will take place. 


A client sends a print job to the print server in the 
standard fashion. Between the client and the print 
server is the firewall, listening for connection attempts 
destined for the print server’s spooling ports. When a 
TCP connection request is made, the firewall intercepts 


IPTABLES -A FORWARD -p TCP -d SSERVER --dport printer -j QUEUE 
IPTABLES -A FORWARD -p TCP -s SSERVER -j ACCEPT 


Figure 10: Initial ruleset for authenticated remote printing. 


struct client_element 
{ 


// together these should indicate a TCP 
32 sourceip; 

__ul6 sourceport; 
ipq_packet_msg_t *syn; 


connection 
//source address 
//origin port on 


of client 
client 


connection 
this 


//syn packet for 
//will challenge 


// timeouts are needed to allow reuse of ports 


struct timeval lasttime; 


// authentication attributes 


long nonce, id; 
unsigned char user [32]; 
int checked; 


// RFC 1179 defines the 
unsigned char 


command protocol 
current_command ; 


// control file is subcommand 02 after " 
unsigned char current_subcomma 


long ctl_length, ctl_ 
long data_length, dat 
int fin; 

// list operators 

struct client_element “prev, *next; 


//last activity time 


//to prevent replay attacks 
//username from challenge 
//control file must match 


//last level 1 cmd rec’d 


receive job" 
nd; //last level 2 cmd rec'd 


consumed; //ctl file length 
a_consumed; //data file length 


Figure 11: lpr packet queuing structure. 
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the syn packet and stores it in a data structure until it 
can confirm the identity of the client. Figure 11 illus- 
trates the data structure used to store client information. 


After storing the TCP syn packet, the firewall 
issues a challenge to the client machine. A specialized 
daemon on the client listens for and responds to these 
challenges. The format of the challenge is simple; it 
consists of a packet id and a random nonce to avoid 
replay attacks. When the client-side daemon receives 
this challenge it constructs a response consisting of the 
same packet id, the incremented nonce, the userid of 
the owner of the daemon (presumably the person sub- 
mitting the job), and the digital signature for the mes- 
sage as associated with the given userid. Figure 12 
illustrates the response issued from the client upon 
receiving a challenge from the firewall. 


packet id 





nonce + 1 


userid 


digital signature 





Figure 12: Structure of the printing challenge 
response. 


When the firewall daemon receives the client’s 
response, it first verifies that the signature on the 
response is valid. To accomplish this step, enterprise 
environments will require the employment of a scal- 
able public key infrastructure. The details of of such a 
system are beyond the scope of our discussion. The 
prototype we present simply assumes that the public 
keys for legitimate users are located in a directory to 
which the firewall daemon has read access. The dae- 
mon searches that directory for a key with the name 
<userid>.pem, where <userid> is extracted from the 
cleartext portion of the client’s response. If the signa- 
ture is not valid for the message, then we do not allow 
the TCP connection request to reach the server at all. 


To enhance the performance of the system and 
reduce unnecessary network traffic, we do not free the 
allocated data structures for the connection right away. 
Because the daemon must only assume a strict con- 
formity to RFC 1179, we do not know what to expect 
from the client in response to the initial connection 
packet being dropped. LPRng version 3.8.19 will issue 
three connection attempts, each timing out at 10 sec- 
onds, before giving up on the connection. Alternate 
implementations of the protocol may use different 
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approaches. Some may even depend solely on the reli- 
ability components of TCP to establish the connection. 
Once the data structures are de-allocated, syn packet 
retransmissions from the client will be seen by the 
firewall as new connection attempts, resulting in a rep- 
etition of the authentication process. We avoid this by 
leaving the data structures in place long enough to 
ensure that the syn packets are re-transmissions and 
not new connection requests; having this information 
allows us to silently discard those packets without 
consuming additional bandwidth and processor time. 
In the prototype implementation, this timeout variable 
is static. For a production release, we would allow a 
configurable runtime option to set this variable, since 
the administrator would have the context information 
needed to optimize it. 


At this point, the firewall has created an instance 
of the data structure illustrated in Figure 11 and loaded 
it with with client IP address/port number combination 
and the verified username of the entity submitting the 
request. By indexing the data structure on host IP/port 
pairs, we can handle inbound packets from a number 
of different connections simultaneously. 


Administrative Jobs 


The LPR protocol allows the superuser signifi- 
cantly more control over job submission and administra- 
tion than regular users. For example, an administrative 
user can issue or remove a print job with a manually 
specified username as the owner of that request. The 
benefit of this is that system administrators can delete 
pending print jobs through the LPR interface without 
having to assume the identity of the actual owner or 
manually modify the state of the printing system. 


This means that for connections submitted by the 
root user of a client machine, the embedded user infor- 
mation may not match the credentials presented during 
authentication of the connection. Suppose that Mary 
needs to print a large document, and the department in 
which she works has a specially designated printer for 
large jobs. She sends the job to the print server with- 
out specifying the correct printer, and the job lands in 
the queue for the default printer, which can’t correctly 
handle the large job. The system administrator notices 
the problematic job, and sends an Iprm command for 
that job to the print server as the root user on Mary’s 
machine. The server deletes the pending job, and 
Mary can be informed that her job was canceled and 
she should submit it to the correct printer. 


If we absolutely require that requests be submit- 
ted in the name of the user whose credentials were 
presented during authentication, we break this compo- 
nent of the protocol. Accordingly, we must make a 
special case for jobs submitted from root users. If the 
challenge response packet contains root as the userid, 
and is properly signed, then we can make no assump- 
tions about what user information the actual job 
request will contain. With a valid signature on the 
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challenge response, we know that the client must be 
an authorized administrative user. Therefore, we place 
no restrictions on what can be done over the lifetime 
of that connection. 


lpr Job Interception 


Once the authentication process is complete (and 
assuming the authenticated user is not root), the 
userspace daemon still intercepts the traffic, because it 
must confirm that job is actually submitted as the user 
who authenticated with the firewall. It observes the 
stream as it passes through, watching for the octet that 
indicates the user field of the control file. The control 
file consists of commands, subcommands, and data, 
exactly one combination of which indicates the user 
for the current job. Specifically, we let everything 
through until we reach command 2 (receive job), sub- 
command 2 (receive control file), at which point we 
begin parsing the control file. 


Each line in the control file has a specific format: 
some character indicates what data will appear on the 
line, then the rest of the line is that data. For our pur- 
poses, we skip lines in the control file until we see one 
that beings with the character 0x0A, which indicates 
the ascii username of the owner of the job being sub- 
mitted. If this user matches the user that previously 
authenticated, we forward all remaining traffic to the 
server. Because there may be additional packets 
already queued in the Netlink buffers, we leave the 
userspace data structures in place and set a flag indi- 
cating that everything else on that connection should 
immediately be forwarded. 


To enhance performance, we also inject two rules 
into the kernel firewall tables. One forwards every- 
thing except the TCP fin packet for the connection 
directly to the server. By bypassing the userspace 
propagation for the remaining packets, we signifi- 
cantly reduce the amortized cost of the authentication 
process. The second rule matches the TCP fin packet 
from the client, indicating the end of the connection. 
This packet is forwarded to the userspace daemon, 
which de-allocates the data structures for this client, 
then forwards the final packet to the server. 


If the user in the control file does not match the 
authenticated user, no further data will be sent to the 
print server, and the connection between the server 
and client will be aborted. Since the job is not actually 
processed until all of the control file is received, the 
document will never print. 


Authentication Failure and Cleanup 


When the firewall denies a print job that has 
failed the authentication process, the server already 
has begun receiving the print job. Once the job is 
denied, these connections must be destroyed so that 
the server can process additional jobs in its queue. If 
we do nothing, eventually, both the server and client 
will timeout and the job will be aborted. The problem 
with this approach is that in the meantime, jobs accu- 
mulate in the queue and are delayed while the server 
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waits for additional data that will never arrive. A bet- 
ter approach is to actively destroy the connection, 
resulting in an immediate abortion of the job on the 
server side. We employ the techniques of Lowth [11] 
to accomplish this. Excerpts from his TCP/IP connec- 
tion cutter software, available under GNU Public 
License, satisfy our requirements that both ends of the 
connection be shut down upon seeing an invalid user 
for a print job. 


Performance Analysis 


We measure the impact the filter has on two 
aspects of the printing system: connections and band- 
width. We present these data separately because there 
is a significant amount of initial overhead on a per- 
connection basis due to the asymmetric encryption 
used for advance user authentication. Once that is 
complete, the filter imposes minimal overhead on data 
transmission. Measurements were conducted from 
both firewalled and control clients, as in the NFS 
benchmarks. 


To calculate job submission times, we modified 
the source code of the print server daemon to record 
the arrival time of the first and last packets on the con- 
nection, and present the difference in a log file. The 
same file was printed from both the firewalled client 
and the control client in these measurements. Submit- 
ting a 17 byte text file from the control client took 
47,719+3,569 uS. As expected, the firewalled client 
achieved a slightly slower rate of transmission, requir- 
ing 202,420+21, 660 US to complete. 
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Figure 13: Firewall performance relative to control 
client. Receipt of control and data files at print 
server. 95% confidence intervals. 


Unlike connection times, which experience a 
small, static delay with respect to file size, we see in 
Figure 13 the impact of channelling each packet 
through the firewall. As files grow in size, more pack- 
ets are required to contain them; each of these packets 
must traverse the firewall, experiencing a small delay 
that is cumulative for submitted job. 
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Future Work 


Having constructed prototypes and described 
potential uses of our system, we now outline specific 
tasks that we intend to pursue in the next stages of the 
project. We present our envisioned final product, com- 
plete with additional examples of applications that can 
take advantage of the services it provides. We are cur- 
rently developing a firewall management toolkit for 
the Netfilter system that uses the userspace queuing 
extensions to allow more active participation from the 
firewall in the system at large. Figure 14 illustrates the 
proposed basic structure of the system. The toolkit 
will include several built-in functions to achieve tasks 
that we believe will be common in applications. Some 
functions will be statically implemented; others will 
be implemented via a pluggable module interface. 
This allows implementations to be swapped out as 
appropriate for specific applications. 












Figure 14: Basic toolkit structure. 
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Figure 15: Toolkit modules. 





Userspace Queues 


Illustrated in Figure 15, we plan to include a 
module interface at both ends of the toolkit in the 
interest of keeping the solution generic. The manner in 
which the system interacts with filtering daemons is 
defined by the library interface modules. Similarly, the 
system’s interaction with the Netfilter queuing system 
is defined by the queue-level modules, though the 
variability in the latter should be less prominent given 
the restrictive interface of the queuing system. 
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Note that services need not be placed behind pro- 
tections provided by the firewall toolkit. They still 
have access to basic services provided by the underly- 
ing Netfilter system, should the administrators deem it 
appropriate. Our proposed toolkit is an extension of 
the firewall model, not a replacement for it. The 
administrator can construct rulesets that direct traffic 
for only some services through the system. Other ser- 
vices can use traditional protections, since only those 
rules that specify the QUEUE target will be propagated 
to the toolkit. 


Planned Toolkit Modules 


The prototypes we have implemented dictate a 
set of useful modules for the toolkit. They provide 
basic functionality that we believe will be common in 
several filtering applications. Here, we outline several 
of the planned toolkit modules. The complete list of 
included modules will be determined as we construct 
the system and determine common needs among 
potential applications. A systems administrator can 
implement and supply additional modules as deemed 
necessary for the desired application. This shields 
application development from cryptographic imple- 
mentations and allows the administrator to focus on 
more abstract functions appropriate to the application 
at hand. 


Cryptographic Support for Authentication 


Since nearly every application we have proposed 
relies heavily on authentication and authorization, we 
must provide direct support for those features in our 
toolkit. Different forms of authentication require dif- 
ferent types of cryptography, so the interface should 
be generic and configurable, providing support for 
arbitrary cryptographic modules. 


Common paradigms such as RSA digital signa- 
tures and fast Blowfish encryption will be included 
toolkit modules in the system we build. Instead of 
relying on high-level generic interfaces provided by 
the standard cryptographic libraries, the administrator 
will have access to a suite of cryptographic functions 
tailored to packet-level encryption and challenge- 
response authentication schemes. 


Connection Destruction 


In most cases where an authentication challenge 
fails, we envision the need to abort any pending con- 
nections to the server. In the NFS case, this is trivial, 
since authentication failure is detectable before a 
client actually connects to the server, and subsequent 
application traffic is sent over the connectionless UDP 
protocol. A failed authentication simply means that 
the firewall blocks all application traffic from the sus- 
picious client, and none reaches the server. 


The lpr example demonstrates a more difficult case, 
however, where authentication cannot be detected until 
after the server connection is in place. Here, we must 
forcefully terminate the open connection to prevent the 
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client from sending any additional data to the server. The 
prototype includes code to accomplish this based on 
[11], but it adds unnecessary complexity to the filtering 
daemon. Such a common function is exemplary of the 
modules our toolkit will include. 


Ruleset Modifications 


Both of the prototypes we preset modify the ker- 
nel firewall’s ruleset over the lifetime of the system. 
The NFS example injects a rule to QUEUE incoming 
mount requests after it learns the port location of the 
file server’s mount daemon. The LPR prototype 
injects a rule to accept all remaining traffic on a con- 
nection if authentication succeeds. The first case is 
functional in nature; the second is a performance 
enhancement. 


Both implementations achieve this via invoca- 
tions of the userspace iptables utility. Since that utility 
interacts with the kernel firewall through system calls 
that modify network sockets, we can achieve the same 
functionality directly. We will include a module that 
provides a library interface to the underlying firewall 
ruleset, allowing direct modification of the rules in 
response to changing system conditions. This will sig- 
nificantly reduce the overhead required to provide a 
dynamic ruleset. 


Implementations and Evaluations 


With a toolkit in place, we intend to revisit the 
two prototypes presented earlier, and re-implement 
them using the firewall toolkit. This will allow us to 
offer a comparison of brute force implementations 
with toolkit assisted constructions. We can compare 
the ease and efficiency of implementations, and ana- 
lyze the differences in performance overhead obtained 
in each method. 


Because we are dealing with a real-time environ- 
ment, performance is of paramount concern. We must 
take steps to ensure that our toolkit introduces minimal 
overhead. In order to achieve this, careful measure- 
ments must be taken on each module that we introduce. 


Conclusions 


We have presented two prototypical applications 
that illustrate how the userspace queuing extensions of 
a commonly available firewall can be used to secure 
legacy protocols. The performance imposition of our 
system is minimal, and certainly justified in light of 
the increased security and functionality the system can 
introduce. We have discussed how this approach can 
be extended into a toolkit that system administrators 
can use to secure additional legacy protocols and 
introduce additional arbitrary functionality at the fire- 
wall. We believe that the toolkit we discuss will allow 
administrators to balance more easily the need to pro- 
vide existing services to users with the need to main- 
tain security features in a networked environment. 
Finally, we have outlined a larger, more general appli- 
cation to which our toolkit will be useful, demonstrat- 
ing the flexibility of our approach. 
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ABSTRACT 


Dynamic coalitions enable autonomous domains to achieve common objectives by sharing 
resources based on negotiated resource-sharing agreements. A major requirement for administering 
dynamic coalitions is the availability of a comprehensive set of access control tools. In this paper we 
discuss the design, implementation, evaluation, and demonstration of such tools. In particular, we 
have developed tools for negotiating resource-sharing agreements, access policy specification, access 
review, wholesale and selective distribution and revocation of privileges, and policy decision and 


enforcement. 


Introduction 


In various collaborative environments such as 
alliances for research and development, health care, 
airline route management, public emergency response, 
and military joint task forces, autonomous domains 
form coalitions to achieve common objectives by 
sharing resources (e.g., objects and applications). 
Coalition resources may be privately or jointly admin- 
istered and resource sharing is achieved by the distri- 
bution of permissions for coalition resources to coali- 
tion members based on negotiated resource-sharing 
agreements, or common access states. These coalitions 
are dynamic in that member domains may leave or 
new domains may join after coalition establishment. 
The focus of this work is on providing tools for 
administering access control in such coalitions. 


Consider the following example of a coalition. Ini- 
tially, three DoD/Intelligence domains form a coalition 
to respond to a threat situation relating to national secu- 
rity (e.g., monitoring transport of nuclear material, activ- 
ities of terrorist groups). Examples of such domains can 
be Defense Intelligence Agency (DIA), National Secu- 
rity Agency (NSA), Defense Threat Reduction Agency 
(DTRA) and Central Intelligence Agency (CIA). These 
three domains decide to share privately owned resources 
(e.g., NSA shares a database of intercepted communica- 
tion and the CIA shares intelligence reports) as well as 
jointly administered resources (e.g., integrated intelli- 
gence data and operation reports). Jointly administered 
resources are typically critical to coalition objectives 
and remain with the coalition even after the departure of 
member domains [16]. Access to jointly administered 
resources is one of the benefits individual domains 
derive from their membership in the coalition. 


To obtain assistance from local authorities for a 
specific threat the DoD/Intelligence domains invite a 


1Author names listed in alphabetical order. 


civilian domain, say the Threat/Emergency Response 
Division of County C, to join the coalition. Upon reso- 
lution of the specific threat, County C leaves the coali- 
tion. Such domain join and leave events should not 
require the breakdown and setup of coalitions from 
scratch. Instead, access control tools must support 
dynamic domain joins and leaves — Dynamic Coalitions 
(DC) — in a seamless manner with ease and efficiency. 


In this example, all four domains are collaborat- 
ing to achieve a common objective (i.e., addressing 
threats to national security) that is desired by all 
domains and achievable by none of them individually. 
To achieve the common objective, each domain has a 
set of resources that are of interest to the others and is 
willing to share some or all of its resources with some 
or all of the other members. 


It is important to note that the domains may not 
be willing to share these resources in the absence of 
this common objective; e.g., the DoD domains may be 
competitors for military tasks and would compete 
based on their available resources. Resource sharing 
by a domain with other domains means that the 
domain (administrator) grants access privileges for the 
resource being shared to the other domains. The shar- 
ing relationships among these domains are defined by 
the access state of the coalition. An access state is 
defined by the permissions each member domain has 
to the shared resources of that coalition. 


Thus negotiating a common access state (CAS) 
means obtaining the agreement of each domain to share 
some of its data and applications with the other domains 
of the coalition. In addition, a CAS? of a coalition may 
include some jointly owned resources created during 
coalition operations, and a common policy for access- 
ing these resources must also be negotiated. 


2A ppendix B lists acronyms used in this paper. 
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The negotiation result is not merely a union of the 
contributed resources necessary to achieve a coalition 
objective. Instead, the negotiated CAS must satisfy 
both resource and permission constraints. In this work 
we consider all constraints that arise in the Role-Based 
Access Control (RBAC) model [8, 9, 10, 2]. For exam- 
ple, when the DoD domains initially form a coalition 
they may agree on a obligation constraint that requires 
at least one user from each domain to have administra- 
tive privileges for jointly administered resources. 


Typically, these constraints arise from coalition 
objectives, access policies that are either jointly or pri- 
vately enforced by autonomous domains, and 
resource-access requirements of coalition applications. 
The specification of negotiation constraints is an 
important part of any access-policy specification and 
drives the negotiation process; e.g., it determines the 
number of negotiation rounds and the convergence to 
and commitment of CAS. Hence, it must be defined in 
a precise manner [17]. 


A typical negotiation for this example would 
begin with the three DoD domains joining the coali- 
tion, specifying negotiation constraints, contributing 
resources that they are willing to share, proposing 
desired CAS, voting on the proposals based on satis- 
faction of constraints and extra-technological prefer- 
ences, and committing to an agreed proposal. This 
negotiation process would be repeated when the civil- 
ian domain joins and leaves the coalition. Since the 
number of objects being negotiated and the number of 
rounds of negotiation could be large even in this small 
example, the negotiation process would be time-con- 
suming and error-prone if undertaken manually. (In 
large coalitions such as those compromising tens of 
domains sharing hundreds of resources [22] this point 
is further emphasized.) Consequently, tools that fully 
(or at least partially) automate the negotiation process 
are needed. Automated negotiation tools would enable 
domain administrators to easily compose proposals for 
a CAS, verify that these proposals satisfy negotiation 
constraints, and commit them to production systems 
once agreed upon. Furthermore, for large-sized coali- 
tions tools that enable administrators to visualize the 
CAS and to administer coalition resources via a graph- 
ical interface are very helpful. 


In addition, access review tools are needed that 
would enable individual domains to review their local 
access state and verify that the state is secure prior to 
negotiation; i.e., that the state satisfies all local domain 
access policies. Based on this review, domains would 
be able to contribute resources to the coalition and 
compose proposals for CAS. Furthermore, access 
review for coalition resources may be undertaken at 
any point in time after coalition setup to (1) view the 
CAS along desired lines (e.g., subject permissions for 
a specific application, application access to specific 
objects), and (2) verify satisfaction of policies and 
constraints (e.g., Separation-of-Duty (SOD) policies, 
obligation constraints). 
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Once the CAS has been negotiated, tools that 
provide wholesale, selective distribution and revoca- 
tion of access privileges are needed. Tools for whole- 
sale distribution/revocation are needed as privileges 
must be granted to/revoked from all users of join- 
ing/leaving domains. Granting/revoking privileges to 
individual users would place undue overhead on 
administrators and, more importantly, would make it 
difficult to support coalitions where duration of 
domain membership could be smaller than coalition 
setup times using today’s deployed technologies (e.g., 
weeks or months). Tools for selective distribu- 
tion/revocation are needed as the tools must selec- 
tively target users of specific domains. One cannot, for 
example, exclude a member domain simply by modi- 
fying CA trust relations because those trust relations 
may be needed to share resources with that domain as 
part of another coalition. 


There exists a need for coalitions in the commer- 
cial world as well as in government settings. What fol- 
lows are two examples of commercial coalitions, 
taken from [15, 17]. First, consider a genetics research 
firm that discovers a gene sequence associated with a 
disease and establishes a coalition with a pharmaceuti- 
cal company, two research hospitals and a Food and 
Drug Administration review board (FDA board) to 
find a cure using the gene sequence. Each domain 
shares some of its local resources with the coalition 
partners to achieve the coalition objective; e.g., the 
genetics firm contributes its gene sequence database, 
the pharmaceutical company provides a drug composi- 
tion tool, the hospitals support clinical trials and give 
access to their patient databases (while preserving 
patient privacy by withholding sensitive patient infor- 
mation such as name, social security number, etc.), 
and the FDA review board shares a database of safety 
regulations. Given the impact of finding a cure, the 
coalition decides to jointly administer an application 
for remote consultation and drug analysis. This 
remote consultation and drug analysis application 
relies on a jointly administered secure group commu- 
nication service. As coalition operations proceed, 
member domains may leave and new domains may 
join the coalition. 


Another example of a coalition would be airline 
companies that collaborate to share various types of 
airlines routes in order to expand their market cover- 
age. Sharing an individual route of a certain type 
implies that the airline domain that owns the route 
grants access permissions required to execute the route 
applications (e.g., reservations, billing, advertising) 
for that route to users of a foreign airline domain. 
Domains share routes (private resources) and may 
choose to jointly administer an auditing application 
that ensures adherence to coalition policies; e.g., poli- 
cies on multi-hop route pricing and frequent flyer mile 
programs. As the coalition proceeds airline partners 
may leave or join the coalition. 
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In this paper we describe a suite of access control 
tools that we have developed for supporting dynamic 
coalitions. The tools employ the RBAC model and 
have been implemented in the Windows 2000 Server 
environment using Java. We have extended? an exist- 
ing RBAC tool, called Multi-Domain Role Control 
Center (MDRCC), to provide access policy negotia- 
tion (i.e., CAS negotiation), specification, and review. 
The extended tool, MDRCC (Multi Domain Role 
Control Center), has been integrated with the Win- 
dows Active Directory for instantiating a negotiated 
CAS and to provide a Policy Decision Point (PDP) for 
resource providers. MDRCC provides a policy visual- 
ization language and a Graphical User Interface (GUI) 
that enables domain administrators to have a common 
view of coalition operations and to administer those 
operations directly from the interface. MDRCC has 
been integrated with an Attribute Certificate Authority 
(ACA) to provide wholesale, selective distribution and 
revocation of privileges. We use the Windows IIS web 
server as an example of a resource provider that 
enforces access policies for shared web pages. We 
have conducted experiments with our tools to show 
that they scale to support reasonable size dynamic 
coalitions and provide results of these experiments. 
We demonstrated the tools at Joint Warrior Interoper- 
ability Demonstration (JWID) 2004 and obtained 
valuable feedback. 


The rest of this paper is organized as follows: We 
discuss related work, present the architecture of our 
system and discuss the salient features on our access 
control tools, describe the implementation of the tools, 
present experimental results, discuss lessons learned, 
and then conclude. 


Related Work 


Access control in distributed systems, in general, 
and in DCs, in particular has been extensively studied 
in the past. However, to the best of our knowledge 
ours is the first complete set of tools for access control 
in DCs that provide (1) support for negotiating access 
policies among domains for coalition set up, (2) 
wholesale selective distribution and revocation of priv- 
ileges, (3) access review capabilities, and (4) support 
for jointly administering coalition critical resources. 
Access Control in Dynamic Coalitions 


The Secure Virtual Enclaves (SVE) [23] infra- 
structure allows multiple organizations to share 
resources while retaining organizational autonomy 
over local resources. SVE provides tools for (1) access 
policy specification via RBAC and (2) policy decision 
and enforcement via Access Calculators and Intercep- 
tor/Enforcers respectively. The SVE infrastructure is 
implemented in Java. Java RMI is used for intra- 
enclave communication and Ensemble — group 


3The initial version of MDRCC, called RCC, provided 
RBAC policy administration for an individual domain and 
was implemented in Visual Basic. 
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communication via JavaGroups Interface is used for 
inter-enclave communication. Resources in SVE are 
distributed applications based on Java RMI, Microsoft 
DCOM, Sun’s Java Web Server and Microsoft’s IIS. 


The Yalta system [24] enables secure tuplespaces 
distributed across administrative domains. A tuple- 
space is a content-addressable shared memory. Entities 
in Yalta share information through a shared tuplespace 
that is distributed across administrative domains. Yalta 
provides for policy specification via Java’s policy 
objects, policy decision and enforcement via Java 
Authentication and Authorization Service(JAAS) inter- 
face, and privilege distribution and revocation via a 
threshold CA and a Certificate Revocation Notification 
(CRN) service respectively. Tuplespaces in Yalta are 
implemented using JavaSpaces. The threshold CA is 
implemented in Java using the Yalta JCE provider, 
which, in turn, is implemented using Jini and Java RMI 
over SSL. 


dRBAC [7] is a decentralized trust management 
and access contro] mechanism. The supporting infra- 
structure is implemented in Java and enables resource 
sharing across administrative domains. The tools pro- 
vide for policy specification via RBAC, and provide 
privilege distribution and revocation and _ policy 
enforcement via a distributed network of wallets. 


However, none of the above work (SVE, Yalta 
and dRBAC) provide tools for negotiating access poli- 
cies, wholesale and selective distribution and revoca- 
tion of privileges, reviewing access policy and joint 
administration of resources. 


Phillips, et al. [21], describe a security model and 
enforcement framework that controls access to APIs 
of software that operate in a distributed environment 
running middleware like JINI or CORBA. The frame- 
work provides tools for policy specification via 
RBAC, policy decision and enforcement and privilege 
distribution and revocation via Unified Security 
Resource (USR). USR is a set of middleware 
resources (JINI and CORBA) that manage all manda- 
tory access control (MAC) and RBAC meta-data for 
users, user roles and resources. However, the frame- 
work uses password based authentication that might 
cause longer coalition set up times. This work pro- 
vides tools for access review but does not provide 
tools for negotiating access policies, jointly adminis- 
tering resources or wholesale selective distribution 
and revocation of privileges. 

TrustBuilder [25] addresses the problem of cre- 
dential negotiation for trust establishment between 
clients and resource servers. This work assumes extra- 
technological resource sharing agreement between 
domains and deals only with credential negotiation. 
This would not scale well in large coalitions where 
users have to negotiate credentials on every access. In 
our work we facilitate multi-party negotiations 
between domains to come up with resource sharing 
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agreements and policies for member domains to gain 
access to those resources. 


Access Control in Distributed Systems 


Though the following work doesn’t deal with 
coalitions explicitly, it is relevant to our work. 
Herzberg et al. [11] presents a Trust Policy Language 
(TPL) that allows organizations to define policies that 
map users to roles based on credentials presented by 
the user and those automatically retrieved by the sys- 
tem from credential repositories. RT [19] is a family 
of Role-Based Trust Management languages for repre- 
senting policies and credentials in distributed autho- 
rization. KeyNote [4] is a decentralized trust manage- 
ment system that provides policy specification via Pol- 
icyMaker, policy decision via Keynote policy inter- 
preter that takes the application’s policy and creden- 
tials presented in the request and outputs a decision. It 
de-couples the application logic from access policy 
and authorization logic. The RBAC model employed 
by RCC may have fewer features when compared with 
TPL, RT, KeyNote, and dRBAC. However, RCC has 
an implementation that provides support for easy con- 
straint specification and verification and an intuitive 
GUI for policy specification, visualization and admin- 
istration. 


Architecture and Access Control Tools 


In Figure 1, we outline the architecture for administra- 
tion of access control in a dynamic coalition compris- 
ing of two domains — which can easily be extended to 
n domains. We assume that each autonomous domain 
has its own Identity Certificate Authority (CA) that 
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distributes identity certificates to users registered in 
that domain, and a container of users, resources 
(objects, applications), and privileges. To participate 
in the coalition each domain (administrator) installs 
the Coalition Resource Management (CRM) toolkit 
that consists of MDRCC, an Attribute CA, and a 
Secure Group Communication (SGC) _ toolkit. 
MDRCC provides specification of access control poli- 
cies in the RBAC model, partially automated negotia- 
tion of the CAS, a visual common view of coalition 
operations, and access review capability. Once the 
CAS is instantiated on the container of users and 
resources, MDRCC provides a PDP via the container 
and administration of access policies for the users and 
resources via an intuitive GUI. The Attribute CA dis- 
tributes (and revokes) attribute certificates (ACs) to 
both local and foreign domain users authorizing access 
to local domain resources. To jointly administer coali- 
tion resources, member domains establish a Joint 
Attribute Authority (JAA) that consists of an MDRCC 
module and a Joint Attribute CA (JACA), that is, an 
Attribute CA whose private key is shared among the 
member domains in a threshold manner [5]. As an 
example of a resource provider, coalition domains also 
establish local domain and coalition web servers that 
manage local and jointly administered applications 
(e.g., App O). The SGC component enables domains 
to communicate securely; e.g., to establish a CAS. 
Policy Specification 

MDRCC uses the RBAC access control model 
for policy specification. We chose the RBAC model 
for two reasons. First, it simplifies administration of 
access control for large systems by separating the 
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Figure 1: Architecture for access control in dynamic coalitions. 
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assignment of resource privileges to roles and users to 
roles [6]. Second, it supports easy specification and 
verification of constraints; e.g., Separation of Duty, 
and obligation constraints [8, 9, 2]. Though there are 
several policy specification tools based on RBAC 
(e.g., [7, 19, 11]) we chose MDRCC because it has 
been implemented, supports constraint specification 
and verification, and has a policy visualization lan- 
guage as well as an intuitive GUI that allows domain 
administrators to have a common view of coalition 
operations and to administer those operations from the 
GUI (e.g., add/remove users from roles). 


Common Access State Negotiation 


We use MDRCC and the SGC tool for partial au- 
tomation of the process of negotiating a CAS among 
member domain. Administrators use MDRCC’s GUI 
to encode negotiation constraints that can be either 
global, in which case they are known and agreed upon 
by all member domains, or /ocal, in which case the 
constraints may remain private to some member 
domains. Types of constraints include Separation-of- 
Duty policies and cardinality constraints. Each domain 
administrator then uses MDRCC’s intuitive GUI to 
specify access policies of resources they are willing to 
share and provides a view of these resources to other 
member domains. Any domain administrator then 
composes a proposal of a CAS (again using MDRCC’s 
GUI) and sends the proposal to the other member 
domains. On receiving this proposal, domain adminis- 
trators can view it and verify that it satisfies both 
global and local domain constraints (local constraints 
are also encoded using MDRCC’s GUI). Domain 
administrators either register their vote on a given pro- 
posal or propose alternative CAS. Once all domains 
agree on a given proposal it is committed in that all 
domain administrators use MDRCC to instantiate the 
negotiated CAS on the local system; i.e., abstract 
objects and permissions in the CAS are mapped to 
actual objects and permissions of target systems. 


Several approaches (e.g., agent-based) that fully 
automate multi-party access policy negotiations have 
been proposed (e.g., [3]) but have not yet been fully 
implemented. Though we are exploring such 
approaches, for the current system we chose the par- 
tially automated approach with MDRCC because, in 
practice, domain administrators would need to get 
involved in the negotiation as they would be responsi- 
ble for administering the coalition and MDRCC pro- 
vides these administrators with an intuitive GUI for 
the negotiation and administration. 


Policy Decision and Enforcement 


When a resource provider gets an access request 
it needs to verify the request against a specified access 
policy. To do so, the resource provider sends a request 
to a PDP and obtains the policy decision. In our solu- 
tion this PDP is provided by the container where 
MDRCC instantiates the negotiated CAS. (As we will 


Administering Access Control in Dynamic Coalitions 


see later, Windows Active Directory is the container in 
our implementation that allows resource providers to 
send Lightweight Directory Access Protocol(LDAP) 
queries for policy decisions). This approach is scalable 
and is employed by many large-scale distributed sys- 
tems; e.g., the XACML specification for web services 
[20]. After the resource provider obtains the policy 
decision it must enforce it. In our system we use a web 
server as an example resource provider and after 
obtaining a policy decision from the PDP the web 
server enforces it by allowing or denying access to the 
requested web page. 


Distribution and Revocation of Privileges 


Once a CAS is negotiated and instantiated, access 
privileges sanctioned in the CAS need to be distributed 
and we use attribute certificates to do so. We assign 
role memberships in attribute certificates and use a 
Public Key Infrastructure (PKI) for distribution and 
revocation of attribute certificates. Coalition users send 
requests to domain Attribute CAs for attribute certifi- 
cates that grant them privileges to access the domain’s 
resources. These requests includes the users’ identity 
certificates. Attribute CAs verify the validity of the 
identity certificate and send requests to the PDP to ver- 
ify the user’s certificate request; i.e., to verify that the 
user has been assigned to the requested role in the 
CAS. Based on the reply from the PDP, Attribute CAs 
issue the attribute certificates. When users send access 
requests to resource providers with their attribute cer- 
tificates, the resource providers verify the validity of 
the attribute certificates and query the PDP to verify if 
the roles assigned in the attribute certificates have the 
necessary privileges to access the requested resources. 
To revoke privileges, Attribute CAs simply revoke the 
attribute certificates and resource providers will no 
longer be able to validate the attribute certificates 
accompanying the access requests. 


In addition to sharing privately owned resources, 
coalition members also benefit from jointly administer- 
ing certain resources. Examples of such resources 
include intelligence reports, purchase orders for equip- 
ment, and financial data. In some coalitions, jointly 
administered resources may include auditing applica- 
tions that are used to ensure that all domains are adher- 
ing to predefined access policies. These resources are 
typically critical for the coalition objectives and, there- 
fore, must remain with the coalition even after the 
departure of member domains. To ensure the continuity 
of access to jointly administered resources in the pres- 
ence of coalition dynamics, member domains setup a 
JAA. Since joint administration is consensus-based it is 
important to ensure that no single domain should be 
able to unilaterally define and modify access policies 
of a jointly administered resources [16]. To achieve 
this, the JAA comprises a Joint Attribute CA whose 
private key is distributed among the member domains; 
i.e., it uses threshold cryptography with each domain 
maintaining a share of the private key and using that 
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share for generating distributed signatures on attribute 
certificates [26]. The process of distributing and 
revoking privileges for jointly administered resources 
is similar to that for private resources except that sig- 
natures on the certificates are computed in a distrib- 
uted manner with all domains participating to ensure 
consensus. 


MDRCC is integrated with the domain Attribute 
CAs in each domain and with the Joint Attribute CA 
at the JAA to provide wholesale, selective distribution 
and revocation of attribute certificates. Administrators 
grant privileges to all users of a joining domain by just 
instantiating the CAS via MDRCC. MDRCC updates 
the container that acts as a PDP (w.r.t. certificate 
issuance) for the Attribute CA. Users of the joining 
domain can then request the Attribute CA (of the 
domain administering the shared resource) for 
attribute certificates before their first access to the 
resource and use the issued attribute certificate to 
access the resource there after. Whenever a user is 
issued an attribute certificate the Attribute CA regis- 
ters it with its local MDRCC. Domain administrators 
can revoke all privileges of all users of a leaving 
domain just as easily as revoking the privileges of an 
individual user — with just a few clicks. When an 
administrator revokes user privileges via MDRCC’s 
GUI, a request is sent to the local Attribute CA to 
revoke the respective attribute certificates. The 
Attribute CA revokes the attribute certificates and 
publishes a CRL. 


Access Review 


MDRCC uses the capabilities of the underlying 
RBAC language and a user-friendly GUI for access 
review and visualization. Domain administrators can (1) 
perform an access review along the desired lines (e.g. 
per-subject review, per-object review), and (2) verify the 
satisfaction of policies and constraints (e.g., Separation- 
of-Duty policies, obligation constraints) using MDRCC’s 
interface. Also, once a policy is specified in MDRCC 
using the interface, MDRCC continuously monitors all 
future administrative actions for compliance. 


Component Design and Implementation 


Our tools are primarily implemented in Java and 
have been tested in Windows 2000 server environ- 
ment. In this section we describe the implementation 
of the main components of our tools and the interac- 
tion between those components. In order for all the 
above components to inter-operate in a coalition envi- 
ronment, inter-domain trust relations must be estab- 
lished. For example, for MDRCC to verify signatures 
on CAS before instantiating it, MDRCC needs to have 
public-keys/certificates of the administrators and also 
trust them. These trust relations are set up manually 
before coalition set up using Windows Certificate 
Trust Lists that are easily composed and shared among 
the various components. 
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Multi-Domain Role Control Center (MDRCC) 


MDRCC is a Java implementation of a Role- 
Based Access Control (RBAC) model [1] extended 
with general role hierarchies across multiple domains, 
static separation of duty and cardinality constraints, 
and advanced access review facilities. In MDRCC, 
each user or role is owned by a particular domain and 
the roles owned by a domain form a hierarchy based 
on the specified role inheritance relation. In each 
domain, the base of the role hierarchy is a special role 
called the domain base role. Users of any domain can 
be assigned to a role in a given domain. Roles are 
assigned abstract permissions to abstract objects. In 
addition, MDRCC includes data structures for (1) 
mapping selected portions of a role hierarchy within a 
domain to user accounts and groups on the domain 
controller, and (2) mapping the authorization informa- 
tion at the enterprise level (that is in terms of abstract 
objects and abstract permissions) to actual objects and 
permissions on the resources resident in various het- 
erogeneous systems in the format required by native 
access control structures. 


MDRCC is a three-tiered component that is 
made up of the following tiers: a Presentation Layer, 
an Application Logic Layer, and a Data Layer. The 
presentation layer comprises the MDRCC client(s), 
the application layer comprises the MDRCC server 
and MDRCC agents (resident in various target sys- 
tems), and the data layer comprises the data reposi- 
tory. An MDRCC client provides a GUI for displaying 
the multi-domain RBAC model graph (or role graph), 
capturing user (Administrator) actions and sending 
them to the (remote) MDRCC server. The MDRCC 
server receives user (administrative) commands from 
the client, and executes them by accordingly updating 
the data layer. The MDRCC server is also responsible 
for mapping selected sub-graphs of the role graph 
(called views) to user accounts and groups on hetero- 
geneous hosts (called also target systems), and for 
mapping abstract objects and role permissions to 
actual objects and permissions structures (e.g., ACLs) 
on those hosts. For these tasks, MDRCC uses agent 
software running on each host to create/delete groups 
and user accounts, and set up ACLs, according to 
commands received from the MDRCC server. The 
data layer consists of a directory service, which stores, 
retrieves, and protects the actual multi-domain RBAC 
data; i.e., the domain information, the user and role 
sets, the various relations, the abstract objects, and the 
mappings between multi-domain RBAC data and tar- 
get system data. In our system, the Windows 2000 
Active Directory provides the directory service; i.e., it 
is the data layer. The communication between 
MDRCC client and server is via SSL and is imple- 
mented using JSSE packages. 


An MDRCC client not only provides visual access 
policy representation (see Figure 2) by displaying the 
role graph but also provides a very intuitive and user 
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friendly GUI for administrative actions; e.g., creating 
roles, assigning permission to roles. For negotiating a 
CAS, domain administrators use an MDRCC client to 
build a role graph (a CAS proposal) from the known set 
of resources that domains are willing to share and export 
the role graph into a compact, machine readable access 
state file. Other domain administrators then import the 
proposed CAS file and visualize it. They can verify that 
the proposed CAS satisfies local permission-based con- 
straints using MDRCC’s constraint verification facilities, 
and then either vote upon it or propose a new CAS. 
MDRCC currently supports the specification and verifi- 
cation of Static Separation of Duty (SSD) and cardinality 
constraints [8]. 


Once a CAS has been agreed upon all the admin- 
istrators sign it with their private keys; i.e., to enforce 
agreement. The signed CAS is then imported into 
MDRCC at each domain and also at JAA. The CAS is 
then instantiated; i.e., abstract objects and permissions 
in the CAS are mapped to actual objects and permis- 
sions of target systems after verifying the signatures 
on the CAS. 


Figure 2 shows a typical view of MDRCC’s GUI 
and the Attribute CA GUI. MDRCC represents users 
and roles of the CAS as ovals of different colors, each 
color corresponding to a different domain. Roles are 
represented with solid ovals and users with solid ovals 
enclosed in an outer oval. Here we see two users from 
county C and a user from domain] assigned to role 
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“Manager”. By right-clicking on any oval a drop- 
down menu is displayed which allows different opera- 
tions for manipulating the MDRCC graph, setting and 
viewing role and user permissions, revoking user and 
viewing Separation-of-Duty constraints. Further 
options are available from the menu bar at the top 
which allow for changing the domain view, so that any 
domain can have a view of the entire CAS. 


For access review administrators can perform 
either a per-subject review (reviewing all the permis- 
sions a subject has to all objects) or a per-object 
review (reviewing all users who have access permis- 
sions to a given object) with just a few clicks. For 
example, by right clicking on a user node in the graph 
and clicking “view permissions” in the drop down 
menu performs a per-subject review. Figure 3 shows 
how MDRCC allows for easy per-subject review of 
privileges. By left-clicking on the oval of a particular 
user, MDRCC displays all of the user’s memberships 
to roles across all domains of the coalition. By right- 
clicking on each role and selecting view permission 
we can see the resources the user gets access to 
through that particular role and also the operations the 
user is allowed to perform on the resource. In some 
cases there is a need to view all of the user’s accesses 
to coalition resources directly without viewing his role 
memberships, and MDRCC allows for viewing those 
accesses by right-clicking on the user name and select- 
ing view permissions. 
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Figure 2: General view of MDRCC and Attribute CA during coalition operation. 


19th Large Installation System Administration Conference (LISA ’05) 


255 


Administering Access Control in Dynamic Coalitions 


Figure 4 shows MDRCC’s interface for viewing 
and defining static Separation-of-Duty (SSD) permis- 
sions. The upper left quarter shows a list of defined 
constraints. A constraint is defined by a role set and a 
threshold. In this case we have one constraint with a 
role set called ssd/ and a threshold of nvo. When a 
constraint is selected from the list, the roles that the 
role set of this constraint encompass are shown in the 
upper right quarter. The meaning of the constraint, 
ssd1:2, shown in the figure is the following: no more 
than two roles of the role set ssd7 can have a common 
user. In Figure 3 we can see that this constraint is vio- 
lated as user J_Doe@domain3 belongs to more than 
two roles of the role set ssd7 and MDRCC shows an 
appropriate error message and does not proceed fur- 
ther until the conflict is resolved. On the lower right 
quarter of Figure 4 are the coalition roles, for selection 
when creating new constraints. 


Certification Authorities 


Our system comprises three different kinds of 
CAs namely, domain Attribute CA, Joint Attribute CA 
and domain Identity CA. We assume that every coali- 
tion member has an Identity CA prior to coalition for- 
mation and hence do not deal with it here. However, 
for our testing and experiments we configured the CA 
service in windows 2000 server environment to act as 
an identity CA for member domains. 


Attribute CA. Each coalition member domain 
has an Attribute CA that issues attribute certificates to 
both local domain and foreign domain users authoriz- 
ing access to local domain resources. Attribute CA is a 
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three-tiered component comprising the following tiers: 
a Presentation Layer, an Application Logic Layer, and 
a Data Layer. The presentation layer comprises the 
Attribute CA console (GUI), the application logic 
layer comprises the CA logic, and the data layer con- 
sists of key stores, issued certificates, published Cer- 
tificate Revocation Lists (CRLs) and persistent data 
structures. The Attribute CA console provides an 
interface for administrative actions (e.g., starting the 
CA, stopping the CA, generating new keys, viewing 
currently issued certificates, viewing revoked certifi- 
cates) and is implemented using javax.Swing package. 
The Application Logic Layer deals with issuing cer- 
tificates, revoking certificates, and generating keys 
and is implemented using the Bouncy Castle Crypto 
API [13] and Java JCE packages. 


Joint Attribute CA. Joint Attribute CA resides at 
JAA and issues attribute certificates to coalition users 
authorizing them access to jointly administered 
resources. Similar to Attribute CA, Joint Attribute CA 
is a three-tiered tool that comprises a Presentation 
Layer, an Application Logic Layer, and a Data Layer. 
But unlike Attribute CA, which uses a standard RSA 
cryptosystem, Joint Attribute CA uses a shared-RSA 
cryptosystem [5] and hence its Application Logic 
Layer and Data Layer are distributed among the coali- 
tion member domains. (In shared public-key cryp- 
tosystems the public key is owned by multiple princi- 
pals with each principal having a share of the corre- 
sponding private key. The shared private key is gener- 
ated in a distributed manner by all the participating 
domains (principals) and each domain retains a share 
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Figure 3: Per-subject review capability of MDRCC. 
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of the private key). These layers consist of (1) signa- 
ture servers that reside in each domain for maintaining 
shares of the private key and signing certificates and 
CRLs and (2) a coordinator that resides at the JAA for 
coordinating the operations of the signature servers; 
ie., the signature operation is distributed and is per- 
formed by the signature servers but is composed into a 
valid RSA signature by the Joint Attribute CA coordi- 
nator (see Appendix A for details on the employed 
shared-cryptosystem). The application logic layer is 
implemented using the Bouncy Castle Crypto API 
[13] and the Yalta JCE provider [24], which imple- 
ments Boneh and Franklin’s shared-RSA cryptosystem 
[26] with signature servers that communicate over 
Java RMI. We augmented the signature servers with a 
policy module (i.e., to enable policy checking capabil- 
ities). The Attribute CA/Joint Attribute CA GUI in 
Figure 2 displays the issued and revoked attribute cer- 
tificates issued to both local users (in case of Attribute 
CA) and users from other domains. It also has menu 
options that allow the operation of Attribute CA to be 
suspended and started as needed. 


Certificate issuance for Attribute CA and Joint 
Attribute CA is provided via a web server. Users 
authenticate to the web server using their domain iden- 
tity certificates and upload their certificate request files, 
which are PEM encoded PKCS 10 files. Users then 
download issued PEM encoded X.509 V3 certificates. 
Both Attribute CA and Joint Attribute CA check with 
local MDRCC (which acts as a PDP for certificate 
issuance) and register issued and revoked certificates 
with MDRCC. In Joint Attribute CA the signature 
servers are also capable of checking certificate requests 
with local MDRCC to ensure that they comply with the 
CAS. Communication between the CAs ( Attribute CA 
and Joint Attribute CA, both coordinator and signature 
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servers) and MDRCC is via LDAP. When an admin- 
istrator instantiates a negotiated CAS on the Active 
Directory, a PDP is automatically created that allows 
both Attribute CA and Joint Attribute CA to verify cer- 
tificate requests providing wholesale distribution of 
privileges. If the instantiated CAS includes the depar- 
ture of a domain then the administrator can revoke all 
users of that domain using the MDRCC client’s GUI. 
This action results in the MDRCC Server sending a 
signed list of serial numbers (of certificates) that need to 
be revoked to the Attribute CA/Joint Attribute CA. 
Attribute CA/Joint Attribute CA process the request and 
publish a CRL on the web server, the URL for which is 
included in all certificates issued by the CAs. Attribute 
CA/Joint Attribute CA then register the revoked certifi- 
cates with MDRCC via LDAP. Thus MDRCC and 
Attribute CA/Joint Attribute CA provide wholesale, 
selective distribution and revocation of privileges. 


Figure 5 shows MDRCC’s interface that allows 
for selective privilege revocation. Any domain (admin- 
istrator) can selectively revoke the users from another 
domain (or all other) in a single operation. This figure 
in particular shows the MDRCC’s ability for selective 
revocation where the privileges of users from another 
domain are revoked only for a specific role and not all 
roles in general. 


Resource Server 


In our implementation we use web pages as 
example resources and use the Windows IIS 5.0 web 
server as the resource server. To access web pages 
users submit attribute certificates to the web server, 
which checks the validity of the certificates by verify- 
ing the signature, checking the expiry time, and check- 
ing against the CRL for revocation information. After 
verifying the certificate (using the certificate verifica- 
tion functions built-in with IIS 5.0), the web server 
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Figure 4: RCC’s interface for specifying Separation-of-Duty constraints. 
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queries MDRCC (via LDAP) and grants or denies the 
request based on the authorization reply from MDRCC. 


Secure Group Communication (SGC) 


Secure group communication is needed in 
dynamic coalitions to aid administrators in negotiation 
during coalition set up and for smooth operation of the 
coalition after it is set up. Since the SGC requirement 
is not unique to dynamic coalitions we chose to use 
existing SGC tools. We use both Secure Spread [14] 
and Secure E-mail List Service [18]. The former pro- 
vides secure, reliable multicast while the latter pro- 
vides ease of use of e-mail in a secure manner. 


Experimental Setup and Results 


We set up experiments to measure the time taken 
for setting up a coalition of three domains and for 
domain join and leave events using our access control 
tools. Each domain in the experiment has ten applica- 
tions that it shares with the coalition (private 
resources), ten roles that have permissions for the 
shared applications, and fifty users that get assigned to 
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foreign domain roles and jointly administered roles for 
access to shared applications. Coalition members set up 
a JAA with ten jointly administered applications and 
ten roles with permissions for these applications. Coali- 
tion setup time is shown in Table | and includes time 
for importing a negotiated CAS (we do not include the 
time for negotiating a CAS as it is largely influenced 
by extra-technological decisions and actions), shared- 
key generation (since distributed shared-key generation 
is probabilistic the key generation times shown in the 
table are averaged generation times averaged over 
10-15 runs), and certificate generation and distribution. 


For coalition dynamics, we measured the time 
taken for a domain to join an existing coalition of 
three domains and for a domain to leave a coalition of 
four domains. The results are shown in Table 1. In 
case of domain departure we assume that the departing 
domain relinquishes its share of private key and hence 
there is no need to generate a new key. Note that this 
assumption only holds as long as the number of mem- 
ber domains that have left the coalition after the last 
key generation is not greater than floor(n/2) where n is 
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Figure 5: MDRCC’s interface for wholesale, selective revocation. 
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number of domains in the coalition at the time of key 
generation. Otherwise the member domains that left 
the coalition can collude to compromise the private 
key [5]. A domain leave event with key generation 
takes about the same amount of time as a domain join 
event. As can been seen from the measured times 
coalition set up and dynamics take only few hours at 
most for moderate sized coalitions there by illustrating 
the scalability of our access control tools. 


Lessons Learned 


The tools were demonstrated at various DARPA 
Principal Investigator (PI) meetings and most recently 
at the Joint Warrior Interoperability Demonstrations 
(JWID) in June 2004 [12]. JWID is an annual Chair- 
man of the Joint Chiefs of Staff event that enables U. S. 
Combatant Commands and the international community 
to investigate command and control, communications 
and computer (C4) solutions that focus on selected core 
objectives. JWID 2004 assessed capabilities and tech- 
nologies that allow information sharing with Homeland 
Security and Homeland Defense partners. This particu- 
lar demonstration experience gave us a lot of useful 
feedback on our tools. JWID 2004 was conducted in a 
simulated operational environment. Our goal was to 
train users to use our tools who, in turn, demonstrated 
the usability of the tools to military personnel. 


Our training experiences can be summarized in 
the following four lessons. First, we could train users 
(who had little security knowledge) in a very short 
amount of time to use our tools because 1) the tools 
enable visual representation of the CAS and provide 
an intuitive GUI and 2) we had generated and pro- 
vided the users with extensive documentation in the 
form of task guides that take the users through execu- 
tion of common coalition administrative operations 
step by step with the aid of images and animations. 
Second, the lack of security knowledge combined with 
multiple windows/interfaces of our tools intimidated 
the users. The users were expecting one unified tool 
(interface) but in reality we have a set of tools that 
integrate functionally and have individual interfaces. 
Third, in spite of our best efforts to test the system 
prior to the demonstration we still over looked flaws 
that were (inadvertently) detected in user training. For 
example, we did not prevent against multiple instanti- 
ations of MDRCC and users started multiple instances 
of MDRCC leaving the data layer in unknown states. 
Fourth, though our set of tools demonstrated the func- 
tional capabilities expected of them by the evaluators 
at JWID 2004, they were not as effective as we would 
have liked largely because users of the tools need to be 
knowledgeable of the underlying technologies in order 
to use them effectively. Our efforts in making them 
user friendly and providing good documentation 
helped us to a certain extent but did not eliminate the 
necessity to educate the users about the underlying 
technologies. 
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Conclusions 


We have developed tools for administering 
access control in dynamic coalitions. In particular, 
tools for negotiating resource-sharing agreements, 
access policy specification, access review, wholesale 
and selective distribution and revocation of privileges, 
and policy decision and enforcement. Our experiments 
demonstrate that these tools scale to handle reasonable 
size coalitions. Demonstrations of the tools at JWID 
2004 provided an opportunity for user training and 
obtaining useful feedback. Though we demonstrated 
our tools in Windows environment they can easily be 
ported to other platforms. In the future, we plan to 
implement the tools over web services for platform 
independence and rapid prototyping of enhanced fea- 
tures and capabilities. 
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Appendix A: Shared RSA Public-Key Cryptosystem 


In this section we discuss the use of shared pub- 
lic key techniques for generation and distribution of 
attribute certificates granting access to jointly owned 
resources. 
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Shared RSA Public Key Generation Algorithm 


Here we review some of the features of the 
shared RSA _ public-key generation algorithm of 
[5].The algorithm enables n domains to generate a 
modulus N = pq and exponents e and d. At the end of 
the computation all domains are convinced that N is 
the product of two primes, however none of them 
know the factorization of N. The public exponent e is 
made public while d is shared among the domains in a 
way that enables m-out-of-n threshold signature gener- 
ation. That is, m domains are able to issue a certificate 
without reconstructing the key d. This also implies 
that an attacker who penetrates at most m-1 domains is 
unable to obtain any information about the private key. 
From the point of view of collusion the algorithm is 
(n-1)/2 private. That is, even if (-1)/2 domains share 
the information they learn during the protocol, they 
will still not be able to recover the factorization of NV 
or the private key d. 


Joint Signatures with Distributed Private Key 
Shares 


For joint administration of access policies, the 
public key K,, of Joint Attribute CA (see Figure 1) is 
generated using the shared key generation algorithm 
resulting in private key shares that are distributed 
among all member domains (i.e., a n-of-n threshold 
sharing of the private key K;!,). Once the public-key 
Ky, has been generated, all domains must apply a 
joint signature algorithm with their private key shares 
in order to sign attribute certificates with the private 
key Kj). The joint signature algorithm involves the 
Joint Attribute CA coordinator sending a message (the 
attribute certificate) to all the signature servers (co- 
signers) with the message M to be signed and a key ID 
comprising the hash of N and the public exponent e. 
Each of the co-signers then apply their corresponding 
private key shares d; to compute S; = M“ mod N and 
send the computations back to the coordinator. The 
coordinator then computes the message signature 


S=[][S; mod N. This joint signature protocol is illus- 


i=l 
trated in [26]. Using this joint signature algorithm, the 
domains sign threshold attribute certificates (and 
CRLs) distributed by Joint Attribute CA. The joint 
signature algorithm also works for a threshold sharing 
of the private key; i.e., the private key Kj|, is shared in 
a f-of-7 manner among the member domains. The 
algorithm is similar to the case above except that when 
the coordinator only needs computations from f 
servers and can compute the message signature 


t 
S=[] 5; mod N. 
i=l 
Appendix B: List of Acronyms 
This appendix contains a list of acronyms used in 
the paper. 


ACA Attribute Certificate Authority 
ACL(s) Access Control List(s) 


Administering Access Control in Dynamic Coalitions 


CA(s) Certificate Authority(ies) 

CAS Common Access State 

CRL(s) Certification Revocation List(s) 

GUI Graphical User Interface 

ILS Internet Information Service 

JAA Joint Attribute Authority 

JACA Joint Attribute Certificate Authority 
LDAP Lightweight Directory Access Protocol 
MDRCC Multi-Domain Role Control Center 
PDP Policy Decision Point 

PKI Public Key Infrastructure 

RBAC Role-Based Access Control 

SGC Secure Group Communication 
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Manage People, Not Userids 


Jon Finke — Rensselaer Polytechnic Institute 


ABSTRACT 


Despite the title, this is not about managing people, but rather managing the enterprise data 
about the people, especially in defining the relationship between a person and the organisation and 
controlling functions based on that relationship, or what some people might refer to as identity 
management. 


Single sign-on is an attractive goal for many organisations. When you include parking gates 
and badge readers on building entrances, the problem gets even more interesting. As we expand 
our deployment of wireless access points and publically accessible network jacks, the need to 
require authentication for access to our virtual world grows stronger. With the need for 
authentication, so grows the demands on the systems that provide authentication and authorisation, 
especially in the area of managing who gets access and revoking that access at the appropriate 
time. Concurrently, with the rising interest in physical security of our facilities, the need for 
authentication and controlling access to our physical world is also growing. This also requires 
tools and systems to manage the people and their status and privileges. 


Both of these issues share many common attributes and can be well addressed by merging 
them into a single system to manage people information, and from that, access to the virtual 
(network) world as well as the physical world. By combining these projects, we are able to take 
advantage of the mandate (and administrative support) to identify all of the people on our campus 
to provide physical access control, and so, manage our virtual world. We will also attempt to 
define a somewhat generic or standard methodology for doing this with our particular business 
rules and requirements confined to a few limited and specific areas. 


While the technical issues are challenging, the more daunting task comes with negotiating 
the institutional politics and getting adequate “buy in” from the appropriate departments to 
provide the people and resources willing to operate and use the eventual technical solutions. This 


paper discusses both the social and technical aspects of those solutions. 


Introduction 


There have been a number of systems developed 
over the years to manage user accounts, Moira [12] in 
1988 (part of MIT’s Project Athena), to Accountworks 
[2] 10 years later and many others. In many ways, this 
process is pretty well understood, and many mecha- 
nisms exist for taking a user account and getting it the 
end system, be it Windows 2000 [7] and others [10, 
11, 3]. In A Retrospective on Twelve Years of LISA 
Proceedings [1], the authors identified 23 papers deal- 
ing with account management. Under the section 
Future Research Opportunities they write: 


Surveying account creation practices would help 
identify why no tool has evolved as superior 
despite many papers on this subject. We believe 
this is because of unrecognised differences in the 
requirements at each site. With all of the require- 
ments explicitly described, it should be possible 
to build a universal tool. 


Like the organisations cited above, we here at 
RPI, have been developing our own system (Simon) 
for automatic creation and management of computer 
accounts. Since Simon’s inception in the early 1990s, 
it has evolved to maintain our telephone directory [5] 


and ID card and parking control systems, as well as 
providing a consistent data feed [9] of people and 
directory information to a number of other systems on 
campus. This paper is in part a case study on how we 
addressed both the political and technical issues. 


The basic objective of all of these systems is to 
automatically create and maintain computer accounts 
for all students, employees, contractors, etc. as 
needed. Given a good data feed, this is basically a 
technical exercise, which has been documented in the 
references cited above and many other places. But in 
all of these cases, the differences come from that pre- 
requisite; the good data feed (or feeds), ideally from 
the system of record for that data element. While 
every site is different, | feel that they all share some 
commonality and I hope to outline some approaches 
that may be applicable to your specific site. One of the 
challenges we face, is that for the most part, no one 
outside of the IT department really cares all that much 
about solving this problem. After all, it is IT’s problem 
to solve, and it doesn’t really impact people outside of 
the IT department at all. We found that by expanding 
the scope of our project from just computer accounts, 
to other “people”’ related activity such as the phone 
directory or the ID card system, we were able to get 
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the interest and assistance of some of the key players 
outside of the IT department. 


From a technical standpoint, while I don’t pro- 
pose to describe the universal tool in this paper, I do 
hope to describe a close to universal methodology that 
can be applied to most educational or corporate! envi- 
ronments. As suggested in the above quote, the prob- 
lem is to define a generic enough tool that it can han- 
dle the requirements of different installations, while 
still allowing the easy application of the appropriate 
business rules with minimal changes to the code base. 


For the most part, dealing with data feeds from 
Human Resources (for employee data) and the Registrars 
office (for student data) is well understood. The chal- 
lenge we face is dealing with the “none of the above” 
categories.2 In an ideal world, we would find a way to 
push that maintenance load out on the departments and 
offices. From the IT standpoint, we don’t really have the 
juice (we can’t mandate cooperation) to get departments 
to maintain status information, but by merging the com- 
puter account management with the more general ID 
card and directory processes, we are able to get sufficient 
interest to maintain the status information. 


In this paper I hope to first describe how we 
addressed the political and social issues for imple- 
menting this system. I will discuss some of the 
arrangements that were made, and some of the argu- 
ments that were used to convince people in other 
departments and divisions that they needed to be part 
of this project and provide the operational support to 
make it all work. In addition, I will describe some of 
the philosophy and guidelines we adopted dealing 
with sources and maintenance of information, and 
some of the pitfalls we encountered. 


Moving from the human realm to the technical 
realm, I will next describe a general schema for man- 
aging “people” (and ‘‘department’’) information, and 
given that, then describe a general, adaptable system 
for managing and delegating all of the oddball cate- 
gories of people, and provide code examples that do 
NOT have our business rules built in, but are easily 
adaptable for any site. From this, we derive a general 
mechanism to evaluate and integrate the “status” of a 
person, in an easily adaptable and flexible manner. 
This in turn drives the business needs of the enterprise, 
be it computer accounts, ID cards, directory entries, or 
whatever the access and authentication needs are. 


Obtaining Buy In from Others 


The fall of 2003 brought to the fore, the issue 
that would finally get serious administrative support 


1Although I am starting from a university environment, we 
are also a major employer, with an active HR department. | 
am not going to address the ISP world. 

2Such as contractors, vendors, adjunct faculty, visiting 
scholars, temporary employees, retires, conference guests, 
research collaborators, emeritus faculty, consultants, Ice 
Hockey officials, model railroad club members, etc. 
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behind the unified people status project, and possibly a 
hot button topic just about anywhere, Parking. In our 
case, it was the installation of parking gates. Now it 
really mattered if someone’s ID card expired — they 
would be denied access to our campus. In point of 
fact, there were at least 40 different departments or 
units identified as ‘‘stake-holders”$ in this project. 
The parking lot access project let us bring some focus 
to the project and get a reasonable number of key peo- 
ple into meetings. 


Authoritative Data Sources 


As part of an earlier data feed project, we found 
it useful to identify the authoritative data source or 
“system of record” for each type of data. Despite hav- 
ing both student and employee information in our cen- 
tral administrative database (SCT Banner),‘ it quickly 
became obvious that although most people were there, 
not all people could be found there. Thus after much 
discussion, it was decided that although Banner would 
remain the system of record for student and employee 
data, the official, campus-wide system of record for 
people data was our locally developed Simon system. 
As each additional group or type of person came 
along, we had to identify the “system of record” for 
that part of the data feed. 


Identifying systems of record is an iterative 
process. Once you identify a data element, and who in 
the organisation owns and maintains it, you then need 
to get them to agree to supply you with an on-going 
feed. This may be complicated by then discovering 
that the authority doesn’t really have all the data you 
need. For example, we were finally able to identify the 
authority for building names, by asking the University 
President at an open “town meeting”. This approach 
may not endear you to the lucky selectee, but it does 
help the process move forward. We then discovered 
that although we could get the “official” names for 
buildings, they didn’t have the more common names? 
or abbreviations for buildings. Abbreviations are 
important — we cut the size of our campus directory 
from 220 pages to 203 simply by using building 
abbreviations! We are still working on resolving this 
issue completely. 


Our current strategy for picking up the smaller 
groups and data elements, is when someone asks for it, 
we ask them who is authoritative for that group. For 
example, when the Provosts office asked for a mailing 
list for emeritus faculty, I asked them who was in 
charge of that information. They admitted that they 


3Stake holders included Parking and Transportation, Public 
Safety, Human Resources, Physical Plant, Contracts and 
Grants, the Student Union, the Registrar, the Provost, the Li- 
brary, Sodexho (food service), Purchasing, and a number of 
others. 

4SunGard SCT — http://www.sct.comhttp://www.sct.com . 

5My office is located in the “Alan M Voorhees Computing 
Center,” which everyone refers to as the “Voorhees Com- 
puting Center” or the “VCC”. 
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were responsible, at which point I offered them a tool 
to maintain that list, that would also be able to give 
them mailing lists, interface with the campus mail 
room distribution system and feed into the ID card 
system. They agreed, we did some training and that 
part of the puzzle dropped into place. It was helpful to 
have a tool ready to go when they asked. 


Sharp Edges 


Even with a good data feed, you still have to 
watch out for some differing expectations between the 
users of the system and the mechanical reality. This 
issue was driven home recently at the end of the fiscal 
year. From a database standpoint, a data value is often 
a time/date value. If you have a time component, there 
is almost 24 hours between a termination date of 
06/30/05 and a start date of 07/01/05. Thus, we had a 
bunch of folks who terminated on June 30th just after 
midnight and their new position did not start until the 
following day at midnight. As a result, about 40 peo- 
ple lost their parking access when they shouldn’t have. 
We will be implementing a “look ahead” function that 
when someone’s job status is terminated, we will look 
ahead to see if they are starting a new job within the 
next two weeks. 


Empowering the Process 


Although we had identified Banner as the system 
of record for employee data, and the Human Resources 
department as the maintainer of that data, we discov- 
ered that although Banner does a fine job tracking eli- 
gible employees and payroll, it wasn’t ready to handle 
real time operations. From a database perspective, a 
potential employee is entered into Banner with some 
demographic information, a department affiliation, etc. 
This can take place weeks before their actual start date, 
and not everyone entered here actually starts as an 
employee. So rather than key off of this, we would wait 
until payroll actually entered them and assigned them a 
job and job classification. When all this was doing was 
driving the telephone directory, we could tell folks to 
wait a few days and the new folks would show up in 
the online directory. But when this entry was required 
to issue the person an ID card and a Parking transpon- 
der, waiting a few days was not acceptable. 


Another requirement for employment, was the 
completion of the IRS 19 form. Historically HR had 
problems getting folks to come up to their office, 
which was located some distance from the main cam- 
pus, to actually sign these forms. So we wrote a little 
application for HR staff, and made a deal with them. If 
they would use this application to mark when a new 
employee was “OK”, (and indicate faculty or staff 
status), we would refuse to issue an ID card, or park- 
ing transponder or computer account until HR set the 
flag. With this in place, new employees had to visit 
Human Resources before they could really do much of 
anything. This also encourages the departments to fol- 
low the HR hiring process. 
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The other end of the employment process is 
employee separation. This impacts not only the IT 
world, but many other departments. When an 
employee leaves, the key shop needs to collect their 
keys; if the person was responsible for hazardous 
materials (gas cylinders, chemicals, etc.), someone 
else needs to take custody of it. In addition, separation 
will impact benefits such as pension and health care. 
One recurring problem we had seen, was when depart- 
ments failed to renew employees on fixed term 
appointments, and essentially, their job ran out. In 
order to deal with this and other issues, Human 
Resources formed a ‘Separation Process’’ committee. 
One of the outcomes of that process was an automatic 
process to generate notification to departments of who 
was coming up for separation. By connecting with this 
process at the database level, we are able to synchro- 
nise our electronic world with the “official”? employ- 
ment status. Now if a department ignores these separa- 
tion reports, a lot of things happen automatically. This 
process gave us a better employee data feed, and puts 
more teeth in the HR process. 


Another example of cooperative projects is the 
management of Emeriti information mentioned above. 
The Provosts office agreed to maintain the lists, which 
feed into the ID card and directory systems. The tool 
they use also lets them get mailing lists back, and even 
the ID card photos. 


One of our remaining challenges, is trying to 
eliminate or at least reduce the number of “Director 
Specials”. Sometimes when a new person comes on 
board, “helpful” people high up in the food chain 
would try to grease the skids by arranging for the new 
person’s email account to be ready when they show 
up. In the past, they would call their favourite director 
in the IT division, and get them to set up special com- 
puter account. This would sometimes result in a sec- 
ond account being created once the normal HR 
process went through, to make matters worse, this sec- 
ond account would be the official account, included in 
the department mailing lists, and so on. With wide 
scale use of card readers and parking passes, even if 
the person got their email account, they still couldn’t 
get an ID card, get parking etc. While we might have 
to jump when the CIO says frog, the parking office 
doesn’t! We are gradually convincing these directors 
and VPs, that the best way to get a new person on 
board, is to get the proper paperwork to HR, and from 
then on, the process is automatic and painless. We 
have on occasion walked the paper through HR, and 
the other systems, but even that extra effort is much 
less work than trying to unsnarl manual entries that 
entered the system in the middle. 


Data Sources 


One of our goals in the design and evolution of 
our system, was to let other people and departments 
do as much of the data entry and management as pos- 
sible. Ideally, they would already be doing this work, 
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and we would just be able to tap into their systems. 
Although they may have been hesitant at first, the 
other departments seem to like this approach, if for no 
other reason, is that it makes them clearly the steward 
of their own data and individuals are encouraged to go 
to the “right place” to get their records and status 
cleaned up. Some obsolete fields have been omitted 
from the table descriptions that follow. 


General People Schema 


The key table for handling people in our system 
is aptly named PEOPLE(Figure 1). It has evolved over 
the years, with new fields being added and others 
being made obsolete, or on their way to being phased 
out. As we refine our data model, this table is moving 
back to it’s intended role of identifying people, and 
other information is being moved to other tables. 


The PEOPLE table shows it’s heritage as the 
driver for creating computer accounts, and some of the 
data models of earlier systems used to feed it. One of 
the annoying lacking, is the the absence of a Mid- 
dle_Name field. When upstream systems started pro- 
viding a middle name as a unique field, it was auto- 
matically merged with the first name to be stored here. 
Later on, a Preferred_First_Name field was provided in 
another table, which can be set by the individual for 
use in directories and displays. There is a lot of code 
that know the old way of doing things, and fixing this 
will be a non trivial task (and is waiting for some other 
things to be rewritten). 


Another problem we have, is with the Student and 
Employee flags. These are® used to control the creation 


ID Number 
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and expiration of computer accounts. There have been 
special cases where someone needed an account, and one 
of these flags have been set via other means. However, 
we have run into cases where staff in the field see these 
flags set and think that the person in question is a current 
employee or student. I look forward to the day when 
these flags are gone and we rely on the status values. 


The xxx_UID fields really don’t belong here, and 
they do imply some policy, like restricting people to 
just one student and one employee account, but also to 
Just one guest account. In practice, a student or 
employee should NEVER have a guest account, and 
there are cases where someone may need more than 
one guest account, or these accounts should not be re- 
used, such as for a contractor who works for one 
department, leaves, and returns to work for another 
department. 


In cases where the person in question comes 
from Banner (which is all students and employees), 
we include their ““PIDM” — this is the primary key 
used by Banner. The general practice is that once a 
person is in Banner, all of their identifying informa- 
tion (name, ID Number, SSN, DOB and Gender) 
comes from Banner. The inclusion of the social secu- 
rity number is disturbing to some people, but it has 
proven very useful at the ID desk to identify people 
who are returning, and are already in the system from 
those who are truly new to Rensselaer. 


One of the ongoing challenges of maintaining the 
data, is dealing with people who get entered into the 


®We will be changing this, and using status values instead. 


Primary Key used to identify a person (or entity). Referenced by many 
The family name or last name. 


Prefix for a name — generally unused. 

Formal suffix to name, such as “‘Jr’’, etc. 

Indicates that person gets a student account — being phased out. 
Indicates that the person gets an employee account — being phased out. 
Indicates that the person gets a guest account — being phased out. 

The unixuid of the student account, if any. 

Unixuid of the employee account, if any. 

Unixuid of the guest account, if any. 

ISO Format ABA card number — ID card number. 

Primary key for person identification in Banner. 


other tables. 
Lastname Varchar2(60) 
Firstnames Varchar2(32) | First and Middle Names. 
Prefix Varchar2(8) 
Suffix Varchar2(8) 
Student Varchar2(1) 
Employee Varchar2(1) 
Guest Varchar2(1) 
Student_Uid Number 
Employee_Uid Number 
Guest_Uid Number 
ISO_Number Number 
FAIMS_PIDM Number 
Spriden_Id Varchar2(9) 


Banner. 
Spriden_Activity_Date | Date 


University ID number (Rensselaer ID Number — RIN) — assigned in 


Date of the last activity in the person ID table in Banner. 
Default budget number for non student charges. 
Person’s social security number. 


A “cleaned”’ version of the lastname (all lowercase, with spaces and 


punctuation removed) to aid in searching. 


Budget_Str Varchar2(32) 

SSN_Str Varchar2(9) 

Birth_Date Date Date of Birth. 
Gender varchar2(1) Gendor of person. 
Clean_Lastname Varchar2(64) 

Clean_Firstnames varchar2(32) 


A cleaned version of the firstname. 


Figure 1: People description. 
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system twice. We have a merge tool that helps us shuf- 
fle records between the different versions of a person, 
and marking the ‘“‘bad”’ record so it is not re-used. 


Students, Employees 


For student data, we have a process that compares 
the Registrar’s list of students with our own copy of 
the student list. Along with daily runs, we can update a 
specific student’s records via a web application. We 
had the added challenge, in that we actually have (or 
had) two registrars, one for each of our campuses, and 
they had different practices in marking “active” stu- 
dents. This difference is demonstrated in the discussion 
of the Person_Status view below. Student processing is 
fully automatic now. Student with account, status or ID 
card issues are referred to the Registrar. The Registrar 
also notifies us when they recode large groups of stu- 
dents, so we don’t panic when 1200 students drop out 
suddenly (due to graduation.) 


In a similar manner, employee data is checked 
once a day with the Human Resources information in 
Banner. We can also do updates of a specific individual 
via a web tool. The Banner data for employees isn’t as 
clean as we would like, which required a special tool 
for HR to use to indicate when employees really start. 
Fortunately, this process has provided some benefits to 
HR, that they are quite willing to use the tool 
(described previously). We did add a safety check that 
stops employee processing if more than 10 percent of 
the employees are marked as being changed. 


Non-Traditional Sources 


There are some groups of people, such as emeriti 
and retirees who are already in the system, but no 
longer active employees. Since we knew that these 
people were already in our system, all we needed to do 
was maintain a list of them. As part of our directory 
project, we had a tool available to manage mailing 
lists. Some minor changes gave us a version of the 
tool to maintain lists of emeriti and retirees. These 
mailing lists were then fed back into the system to 
provide additional status entries. 


Guests 
Although we had students and employee status 
well in hand, that still left “none of the above.” With 


the deployment of RFID cards and Parking transponders 
to provide access to campus buildings and parking 
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facilities, the problem of maintaining status information 
for “ID Guests” became very real. Our original practice 
of creating a campus computing account (which as a 
side effect, would generate an ID card number for them) 
was not going to handle this. 


One of the principles that we adopted, the 13th 
amendment to the Constitution notwithstanding, is that 
everyone had to be “owned” by an on campus person 
or department. For regular employees, Human 
Resources plays this role and for students, the Regis- 
trar handles it. But for everyone else, we needed to 
designate an ‘“‘owner”’. In many cases we were able to 
use our existing departmental directory administrators 
to handle this role, and for other cases we created new 
“departments” and assigned administrators there. 
These administrators are assumed to know which 
guests are still with their department and can expire or 
renew them as needed. For some categories of visitors, 
they can also control if that person gets a computer 
account and if that person is to be included in the 
online directory. 


The primary point of contact for guests, is our ID 
card office. The folks there have a tool that lets them 
make entries in the ID_People table (Figure 2). Entries 
here are automatically propagated into the People ta- 
ble, and are also used to provide status values. Some 
of the fields have been omitted for brevity including 
some personal ID fields (SSN, DOB, etc.), but the key 
ones are included. 


Some entries in the ID_People table will have a 
Sponsor specified. The sponsor will point to another 
person, and this used used in the case of dependents or 
spouse, or a personal care worker; someone whose 
relationship with Rensselaer is a direct result of some 
other member of the community. This allows us to 
automatically “expire” dependents when their spon- 
sor is no longer eligible to sponsor dependents. 


If an entry doesn’t have a sponsor, then it will 
have an Affiliation_Id. This points to a department. 
Originally, the department tree for the ID Guest sys- 
tem and the department tree for directory were differ- 
ent. We have since merged them and although we still 
have two tables, the tools keep them synchronised. 
One of the big benefits of merging these two trees into 
one, is that the directory departmental administrators 
can maintain ID guests as well. 





Person_Id Number 

Lastname Varchar2(64) | The last name. 
Firstnames Varchar2(32) 

SSN Number 

Entry_Type_Id Number 

Sponsor Number 

Requestor Number 

Affiliation_Id Number 


Expiration_Date | Date 
Dir_Flag varchar2(1) 


Primary key — matches People.Id 


The first and middle names. 

The SSN or Rensselaer ID Number if available. 

Identifies different types of entries, see Id_Entry_Types. 

The People.Id of the person who “sponsored” this guest, if any. 
The People.Id of the person who requested this entry. 

An identifier of the department sponsoring this entry. 

Date this guest entry expires (can be renewed). 

Flag to control inclusion in the campus directory. 


Figure 2: |D_People description. 
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The ID Card Office staff use the tool to enter a 
new person into the system. The application attempts 
to locate the new person in the existing database, 
searching by name and by ID number. We really want 
to avoid duplicating people. We can still make the 
guest entry, but the Person_Id will match the original 
record. The next step is to select what type of guest 
they are, which then determines the additional ques- 
tions they need to answer. The choices are contained 
in Id_Entry_Types (Figure 3). The xxx_Required flags 
tell the application if it needs to prompt for those 
fields. In the cases of sponsor and requestor, the per- 
son must be in the existing database, and in the case of 
a sponsor, the sponsor’s status must be one that allows 
them to sponsor a guest. (How this is done will be dis- 
cussed later). If the entry type has an affiliation id, that 
tells the application the root of the departmental tree it 
should use for the selection list. This ensures that 
Incubator staff must work for an Incubator company 
and so on. The department administration tool will 
expire all of the members of that department (or other 
organisation) is terminated. Our current list of entry 
types is in Figure 4. 

Other flags will determine further processing. 
The directory flag can put someone into the directory 
automatically with no choice to the departmental 


Entry_Type Varchar2(24) 
Sponsor_Required Varchar2(1) 
Requestor Required | Varchar2(1) 
SSN_ Required varchar2(1) 
Affiliation_Id Number 
Expiration_Delta Number 


Dir_Flag 
Display_Rank 


varchar2(1) 
number(3) 
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administrator. For example, our ROTC staff go in — no 
choice. Other categories such as visiting researchers or 
Incubator staff can go in, but that can be controlled by 
the departmental administrators (“‘O” optional, default 
include, “*P” — optional, but default to not include) 
and the rest don’t go into the directory at all. If there is 
an expiration delta, that number of days is added to the 
current date to get a default expiration date. 


Entry Points, Delegation, Short Term Visitors 


Not everyone on campus is there long enough to 
get their own picture ID card. While they still need 
one for building access and dining plans, the overhead 
of producing ID cards for someone who was going to 
be on campus for three days for a conference was sim- 
ply not worth it. There are a few cases where an ID 
card is desirable as a keep-sake, for the most part, we 
have a collection of generic cards (guest 1-Guest999), 


While that worked for the physical access, we 
still had the problem of visitors to campus who needed 
to authenticate to access our VPN to get on campus, or 
to access the wireless network. This would result in 
the request and creation of a full computer guest 
account, with email access, printing allocations, disk 
charges, and would take a business day or two to get 
set up, or someone in the department would “loan” 
their account out. While the first option was arduous, 


A short name for this type of entry. 

A flag indicating of a sponsor is required for this type of entry. 
A flag indicating of a requestor is required for this type of entry. 
A flag indicating if an ID number is required. 

An identifier of the department sponsoring this entry. 

Number of days from now for entries to expire. 

Flag to help control inclusion in the campus directory. 

A rank order to be used for display purposes. 


Figure 3: ID_Entry_Type description. 








Entry Type 
Special Programs 
Temporary Employee 
Conference Guest 
Off Campus Contractor 
On Campus Vendor 
Department Vendor 
RU Club Member 
Incubator 
Tech Park 
Dependent 
Spouse 
Domestic Partner 
Personal Aid 
Visiting Researcher 
Visiting Faculty 
ROTC Faculty 
ROTC Staff 
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Sponsor 
Required 


Requestor 
Required 





Display Entry 
Type Id 
91121096 
91318569 
91114523 
91393994 
91398702 
91402506 
91405098 
91068656 
91080046 
91449911 
91449912 
91449913 
91449917 
91399849 
91399850 
91397007 
91397008 
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Figure 4: Current ID Entry types. 
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the second could result in the staff member who 
“loaned” the account being terminated. 


Instead, we developed a tool that allows depart- 
mental administrators (and each department has at 
least one, see the next section) to allocate a temporary 
account that will allow someone to access the VPN to 
get in, or the wireless network to get out. When they 
allocate an account, they can specify an expiration 
time from 3 hours up to 2 days (accounts can be 
renewed), after which time, the password will be reset 
automatically. They can also specify a comment for 
that assignment that they can later review. Once allo- 
cated, these accounts “belong” to that department. 
Once they have expired, they are available for re-use 
by that department. If the department has no “free” 
accounts, a new one will be drawn from a pool. The 
pool is monitored to ensure a ready supply of new 
temporary accounts. All of this happens without any 
involvement by the IT staff — it is entirely in the hands 
of the department administrators. 


This gives departmental administrators the abil- 
ity to satisfy some of the needs of visitors to their 
department quickly and easily. 


Departments 


We really can’t talk too much about managing 
people, without understanding what department they 
belong to. One of the key points of this system, is that 
there is someone responsible for maintaining the status 
information for everyone on campus. This may be 
indirect like with HR maintaining employee informa- 
tion, or the Registrar maintaining student information, 
or directly with departmental administrators maintain- 
ing guest information. Although there are a few cate- 
gories such as dependents or spouses that are “owned” 
by another person, all of the rest of the guests need to 
be “owned” by a department. This requires a good list 
of departments, and possibly the relationship between 
departments (all of the engineering departments belong 
to the school of engineering, etc.). 


As part of the telephone directory project, we 
had to get a pretty good handle on departments. When 
the University went to a fund accounting system with 
composite account numbers, one of the elements was 


Students 
Registrar 


Employees 
Human Resources 


Hartford Directory 
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the ORGN (Organisation). These were arranged in a 
hierarchy, with every orgn rolling up to it’s parent and 
so on to the president. Our joy was short lived how- 
ever. The primary purpose of the ORGN tree, was 
financial accounting. This resulted in lots of extra 
“departments” (like copy center), departments that 
were not “data enterable”, so these were paired with 
an “office of XYZ” department so charges could be 
rolled up. To make matters worse, the department 
“names” were limited to 30 characters, and there was 
no place, nor interest from the finance office in main- 
taining abbreviations. 


To get around this, we created our own superset 
of the ORGN tree, where we could add our own “‘vir- 
tual’’ organisations, specify alternate names for real 
organisations as well as abbreviations. An agreement 
was made that we would only create ORGN codes (the 
primary key) that started with “V”, “S” or “DH”, 
and the rest of the name space belonged to them. New 
ORGNs from the controller’s office are automatically 
added to the directory (and the telecom staff double 
checks them — some are not included). But this gave 
us a place to expand the existing university depart- 
mental hierarchy, as well as build new trees for ven- 
dors, and many other special groups. 


Another aspect of the department management, 
was to identify one or more administrators for each 
department. If a department did not have one, it could 
inherit administrators from it’s parent. Although this 
was originally intended for managing the telephone 
directory, it gave is a ready made list of people who 
can take on additional responsibility for guests in their 
department. This was a key component in ensuring 
that status information for EVERY person on campus 
was the responsibility for an identified person (or 
department) to maintain. 


People Status 


Now that we have identified all of our different 
data feeds, it is now time to put everything together 
into a uniform object (Figure 5). The first step is to 
create an Oracle view, Person_Status (Figure 6) that 
combines information from Employees, Students, 
ID_Guests, Directory_Aux_Entries and Hartford_Raw_Dir 


ID Guests 
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Figure 5: People Status data flow. 
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tables. We don’t want to be querying this view all of 
the time, and we do want to know when someone’s 
status changes, and maintain some history. To this end, 
we have a People_Status table (Figure 8). And finally, 
we feed this information to other systems. 


Person_Status 


The person status support was originally devel- 
oped to provide a patron feed for our Library circula- 
tion system. It was later enhanced to feed our ID card 
system. It is now being generalised as people status 
management tool, however, some traces of the original 
library and ID card system remain in the column defi- 
nitions. Most of this information is being moved to 
other tables that join with Person_Status based on the 
Status_ID column. The Person_Status view (Figure 6) is 
the key place where all the different inputs to the sta- 
tus process are brought together. 

The Person_Status view is coupled tightly to Peo- 
ple_Status_Types (Figure 7) to generate a snapshot of 
everyone’s “‘status”’. The view and this table is where 


Finke 


all of the site specific stuff lives. The view goes and 
roots around in a number of different database tables 
to come up with a standard set of status entries. The 
People_Status_Types table has columns for joining with 
the view to determine types, and some other columns 
that are used to feed the Library and some other sys- 
tems. I have omitted this second set of columns, as 
they are gradually being phased out in favour of other 
tables. A subset of the entries in the People_Sta- 
tus_Types is available in the first appendix of the paper. 


The Person_Status view is a series of joins with 
other tables and the People_Status_Types table, that are 
connected together with a SQL UNION directive. For 
ease of formatting, I have broken the view up into indi- 
vidual stanzas (Displays). This is where all the magic 
happens. 


Get Employees 


This first section looks in the employees table to get 
our current employees. The important distinction here is 
really the key 1 looking for “A” (active) employees and 


Patron Type for Library Circulation System 

Identifier for status type. 

‘Status’ entry for use on the ID card. 

Identifier for the person record for this entry. 

Name of the department, if applicable. 

Department Identification code. Used with Coas_Code to 


identify department. 


Department Chart of Accounts identifier. 

Affiliation identifier for non departmental entries. 

Flag indicating if person should be in the fac/staff directory. 
Date when status expires, if known. 


Figure 6: Person_Status description. 


Lib_Patron_Type | varchar2(3) 
Status_Id Number 
ID_Card_Status varchar2(32) 
Person_Id Number 
Orgn_Name Varchar2(32) 
Orgn_Code Varchar2(6) 
Coas_Code Varchar2(1) 
Type_Key Number 
In_Dir varchar2(1) 
End_Date Date 
Status_Id Number 
Rank Number 
ID_Card_Status varchar2(32) 
Person_Id Number 
Status_Category varchar2(8) 
Source_Table Name | varchar2(32) 
Key_l varchar2(16) 
Key_2 varchar2(8) 
Nkey_1 number 
Comments Varchar2(255) 


Identifier for status type. 

Orders status values, highest value is returned. 
“Status” entry for use on the ID card. 

Identifier for the person record for this entry. 
Rough category for status types. 

The name of the table we will match with. 

A key to match against, usage depends on cate- 
gory and table name. 

A second key (there is also a key_3 and key_4). 
A numeric key, usage depends on the status 
category and source table. 

A description of this entry. 


Figure 7: People_Status_Types description. 





select lib_patron_Type, status_id, id_card_status, person_id, 
substr(Ftvorgn_Title,1,32),. pebempl_orgn_home,’9’, 


to_number(null) TYPE_KEY, pst.in_dir, 


e.nbrjobs_end_date 


from employees E, people_status_types pst, fimsmgr.ftvorgn f 
where Status_Category=’Emp’ and key_1 = NBRJOBS_STATUS 

and PEBEMPL_ECLS_Code like nvl(key_2,PEBEMPL_ECLS_Code) 

and pebempl_orgn_home = f.ftvorgn_orgn_code 


Union... 


Display 1: Retrieving employees. 
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key 2 looking for the employee classification. It also 
dives into the Ftvorgn table to come up with the depart- 
ment name.’ See Display 1. 


Get New Employees 


We dive back into the employees table again to 
look for new employees. The employee classification 
doesn’t appear in the employees table until the next 
payroll cycle, and we really don’t want to wait that 
long to start issuing ID cards and the like. So Human 
Resources has a tool to set the HR_Ok and HR_Status 
fields appropriately, and this stanza will pick those up. 
In the course of normal employee record processing, 
these fields will get cleared when the classification 
information comes through so we don’t have people 
with two employee status values. But even if some- 
thing goes wrong, and that isn’t cleared, since the reg- 
ular employee status types have a higher rank, that 
entry will override this entry. See Display 2. 


Get Troy Students 


Continuing with the view, we go after our local 
students. We are getting the same number of columns 
as we did in the first stanza (a requirement of a view), 
but most of the department related fields we are 


7People well versed in Oracle and/or Banner will note that 
column aliases are missing, and additional conditions are 
needed to make this work. These have been removed to aid 
formatting. See the end of the paper for how to view the ac- 
tual, working code. 
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returning as null. We look for active students Sgbst- 
dtn_Status_Code='AS’, and those in the Troy campus 
(Key 1), and also breaking them down by College 
(school) and level (Grad/Undergrad). Although this 
level of granularity may be overkill for most applica- 
tions, the Library wanted this fine grain distinction. 
See Display 3. 


Get Hartford Students 


We have a second campus in Hartford, and they 
code students somewhat differently. In Troy, the Reg- 
istrar aggressively recodes students who are not 
active. But our Hartford campus does a lot of continu- 
ing education, where students just take a course or 
two, so their status is left as ‘AS’. So to handle this 
case, we pull the same columns, but look for some 
other keys in the students table, Registered_This_ 
Term. We actually have a third campus which is being 
phased out, so I will skip that stanza of the view. See 
Display 4. 

Mailing List Entries 


Our method of handling people who are already in 
the system, and need some special status, like what we 
did for the Emeriti faculty, is handled internally by 
adding them to a special “department”. These tools were 
originally developed for the phone directory, and worked 
well here as well. In this case, an entry is made in the 
Directory_Aux_Entries table (which is listed as Dir_Aux_Ent 
in the appendix), and we pick up the appropriate entries. 


select lib_patron_Type, status_id, id_card_status, person_id, 
nvl(substr(Ftvorgn Title,1,32),’Unknown’), pebempl_orgn_home, ‘°9’ 
to_number(null), pst.in_dir, to_date(null) 


from employees E, people_status_types pst, 


where Status_Category=’NewEmp’ 


and hr_ok = ’Y’ and hr_status = key_l 


and nvl(pebempl_orgn_home, ’XXXXXXX") 
Union... 


ftvorgn_coas_9 f 


f.ftvorgn_orgn_code (+) 


Display 2: Select new employees. 


select lib_patron_Type, status_id, id_card_status, person_id, 
Sgbstdn_Majr_Code_1l, Null, Null, to_number(null), Null, To_Date(Nul11) 
from banner_students bs, people_status_types pst 


where Status_Category=’BStu’ 


and Key_1 = SgbStdn_Campus_Code 
and ( key_2 
and Key_3 = Sgbstdn_Level_Code 


and Sgbstdn_Status_Code = ’AS’ 
Union... 


= Sgbstdn_Coll_Code_1l or key_2 is null ) 


Display 3: Select Troy students. 


select lib_patron_Type., status_id, id_card_status, person_id, 


Sgbstdn_Majr_Code_1, Null, Null, 


to_number(null), Null, To_Date(Null) 


from banner_students bs, people_status_types pst 


where Status_Category=’HStu’ 
and Key_l = Sgbstdn_Level_Code 


and ( Hart_Reg This_Term = "Y’ or Registered_This_Term = 'Y’ ) 


and Sgbstdn_Campus_Code = ’H’ 
and Key_2 = Sgbstdn_Status_Code 
Union... 


Display 4: Select Hartford students. 
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We use a different table for the department name this 
time, this stanza was written later and I should revise the 
employee entries. See Display 5. 


ID Guests 


This is the place where we get all of our id 
guests. A number of the id entry types map directly to 
a status type. You can actually compare the Nkey_1 
values in the appendix with the Entry_Type_Id values in 
the Entry Types listing above. We also make a check 
to ensure that the guest entry either does not have an 
expiration date, or that date is some time in the future. 
If there isn’t one on the record, the current time will be 
used, but that is good enough! See Display 6. 


People Living on Campus 


We have a significant number of people who are 
living on campus after they graduate. One of the side 
effects of graduating, is losing your student status, 
which triggers a number of events, including cancella- 
tion of your ID card, which locks you out of your 
building! Although these people were no longer regis- 
tered students, they did have housing contracts. We 
added some information about room assignments to 
our student table, and added this final stanza to the 
Person_Status view. During the normal school year, 
this entry will provide a second status value to all on 
campus students, but since it is of a lower rank, it is 
ignored. See Display 7. 


Finke 


People_Status 


While the Person_Status view will give us a 
snapshot of the current status of everyone, it is not 
able to provide any historical information. A common 
question when checking someone with no status, is 
what was their status before it ended. There were also 
concerns about performance of this view — it is look- 
ing into a number of other database tables. For both of 
those reasons, we have a process that runs daily that 
checks the current status of everyone, with what we 
have saved in the People_Status table (Figure 8). We 
can also invoke this processing for a specific person. 
This function is used for several administrative appli- 
cations when they want to ensure someone’s status 
record is current as of this instant. Refreshing a single 
person’s status is almost instant, it takes longer to 
redraw the screen. 


The People_Status table has the same columns as 
the Person_Status view, with the addition of a start and 
end date, as well as some numeric columns (When_ 
Inserted and When_Marked_For_Delete) to assist with 
data propagation. Changes in a person’s status are also 
reported in the Meta Change Queue subsystem, so 
other systems can easily watch for changes in a per- 
son’s status. 


8On a recent mun, this process compared 19,000 records in 
just over a minute, a performance level I am comfortable with. 


Select lib_patron_type, status_id, id_card_status, person_id, 
substr(nvl(orgn_common_name,orgn_name),1,32), dae.orgn_code, 
dae.coas_code, title_id, dd.def_include, to_date(null1l) 

from Directory_Aux_Entries DAE, people_status_types pst, 


directory_departments dd 


where Status_Category = ’DirAux’ 


and pst.Key_l = DAE.Orgn_Code and pst.Key_2 = DAE.Coas_Code 
and dae.orgn_code = dd.orgn_code and dae.coas_code = dd.coas_code 


Union.. 


Display 5: Select mailing list entries. 


select lib_patron_Type, status_id, id_card_status, person_id, 


ia.name, ia.orgn_code, ia.coas_code, 


nvl(ip.affiliation_id.,ip.sponsor), ip.dir_flag, 


ip.expiration_date 


from id_people ip, people_status_types pst, simon.id_affiliates ia 


where Source_Table_ Name = ’Id_People’ 
and Nkey_1 = Ip.ENTRY_TYPE_ID 


and nvl(ip.expiration_date,sysdate) >= sysdate 


and Key_l = ’CURRENT’ 


and ip.affiliation_id = ia.affiliation_id (+) 


Union... 


Display 6: Select ID guests. 


select lib_patron_Type,status_id,id_card_status,person_id, 


Slrrasg_ Bldg Code || 


Null, Null, to_number(null), Null, 


*-* || Slrrasg_Room_Number, 
Slrrasg_End_Date 


from banner_students bs, people_status_types pst 


where Status_Category=’ResLife’ 
and Slrrasg_Active = ’Y’ 


and ( key_1 is null or key_l = slrrasg_bldg_code ) 


Display 7: Select people living on campus. 
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Making Use of People Status 


The original target of the people status support was 
to feed the circulation system for our Library. It was 
later expanded to feed the ID card system. It became 
obvious that extending the People_Status_Types table for 
every new application was the wrong approach. Instead, 
we added a set of People_Status_Aux... tables that 
allowed us to define new flags for each status type, and 
developed a tool to allow administrators of other sys- 
tems to set their flags and not interfere with other con- 
sumers of the status information. We also use a person’s 
status to trigger other processing. 


People Status Auxiliary 


Adding columns to tables, or creating new tables, 
and then writing interface routines and tools to manage 
them can get pretty tedious, and takes a lot of time that 
could be better spent on other tasks. I wanted to be able 
to “extend” status related function by making table 
entries in the database, rather than writing new code. 


To start, we defined the table People_Status_Aux_ 
Master (Figure 9) that define new “streams” of status 
information. Initially I was envisioning selecting peo- 
ple for data feeds (or streams) to other systems, and 
the name stuck. This defines the name of the stream 
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and lists an Oracle role that will be used for access 
control by the administrative tool. 


The next step was to define a prototype table, 
People_Status_Aux_Proto (Figure 10), to identify the 
possible fields, the field types and the default values. 
This is used by the administrative tool to automati- 
cally generate the appropriate columns and switches 
on the web page. 


The final table holds the actual values for each 
stream, People_Status_Aux_Values (Figure 11) which 
stores a field value for each status, stream, field triple. 


Once we have defined a new stream, assigned it 
an access role, and defined one or more fields for that 
stream, we are able to delegate the management of these 
values to the appropriate interested parties. At present 
we have the following streams defined and in use; see 
Figure 12. 


Now that we have all of this information in the 
system, and control delegated to the appropriate 
offices, we need to get it out again. At the most basic 
level, we have a PL/SQL package with some routines, 
one that will give you a list of everyone with a particu- 
lar Stream/Field/Value triple and another that will 
return true or false, if a given person has a particular 


Person identifier. 

Identifier for status type. 

Date when this status was initiated. 

Date when this status will terminate if known or was terminated. 
Name of the department, if applicable. 

Department Identification code. Used with Coas_Code to 


identify department. 


Person_Id Number 
Status_Id Number 
Start_Date Date 
End_Date Date 
Orgn_Name Varchar2(32) 
Orgn_Code Varchar2(6) 
Coas_Code Varchar2(1) 
Type_Key Number 
In_Dir varchar2(1) 
When_Inserted Number 
When_Marked_For_Delete | Number 


Department Chart of Accounts identifier. 

Affiliation identifier for non departmental entries. 

Flag indicating if person should be in the fac/staff directory. 
Sequence value when record was inserted. 

Sequence value when record was considered to be deleted. 


Figure 8: People_Status description. 


Stream_Name 
Access_Role 


varchar2(32) 
varchar2(32) 








The name of the stream. 
An Oracle role that can manage this stream. 


Figure 9: People_Status_Aux_Master description. 


The name of the stream. Must exist in the Master table. 


The data type — to assist the web tool in formatting and controlling. 


Stream_Name | varchar2(32) 
Field_Name varchar2(32) | The name of the field. 
Field_Type varchar2(32) 
Field_Length | varchar2(32) 


Field_Default | varchar2(32) 


The maximum length of the field where applicable. 
The default value to use. 
Rank order to display fields on the web tool. 


Figure 10: People_Status_Aux_Proto description. 


Field_Rank Number 
Status_Id Number 
Stream_Name | varchar2(32) 
Field _Name varchar2(32) 
Field_ Value varchar2(32) 


The People_Status_Types.Status_Id of the status getting this value. 
The name of the stream. 

The name of the field. 

The value for this particular field. 


Figure 11: People_Status_Aux_Values description. 
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Stream/Field/Value value. We have also set up Gener- 
ate_File [6, 8] targets to extract list based on stream 
and field combinations. We have also written several 
“wrapper” packages to provide the appropriate 
streams to people and applications who need to con- 
nect directly to the database. 


Other Uses 


Our physical access control system (Card read- 
ers, parking gates), also uses status, or more specifi- 
cally the lack of any status to automatically terminate 
all access to campus building, roadways and parking. 
We also use status to control if and how people are 
included in the different campus directories. 


Conclusions 


The people status project has been evolving over a 
number of years here. During that time, some of the 
original data sources have been replaced, and new ones 
have been added. In some cases the direction of data 
flow has changed. At one time the faculty/staff directory 
data drove the status information; that has been reversed, 
status data now drives the faculty staff directory. 


The People Status project at Rensselaer has 
reached critical mass. It impacts enough systems that 
matter to people (such as Parking!) that folks are will- 
ing to play by our rules. And it does make life simpler 
— students enter the system via the Registrar, employ- 
ees get in via Human Resources and everyone else 
comes in via the ID card office. Once in, the assorted 
special cases get picked up by the departments who 
care about those people and things work well and with 
a minimum of human intervention. 


Implementation Lessons 


The people status project was not rolled out as a 
complete package starting at nothing, but rather was 
built on several smaller projects such as the directory 
and computer account management. These systems 
allowed us to identify and refine the data sources and 
procedures, while assembling the infrastructure. It also 
provides time and opportunity to develop the working 
relationships with key players in other departments. 


EBS List 
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There are two types of people I needed to work 
with to pull this project together, people who had data 
I could use, and people who needed data. By starting 
with some smaller projects (they didn’t seem small at 
the time!) like the phone directory and computer 
account management, I was able to put together a 
comprehensive enough “people”? database, that I 
could get people who needed feeds interested. By hav- 
ing tools and techniques readily available to enable 
them to control their feeds (Generate File, Meta 
Change Queue, etc.), | was able to get a number of 
“clients” for ‘‘my data”’. These clients in turn made it 
more attractive for the owners of the data to work with 
me, since it would leverage their work. 


I was also able to target groups that were outside 
of the mainstream, and so didn’t have quite the level 
of IT support that they may have wanted. I help them, 
and they are willing to modify their procedures and 
processes to better accommodate what I need. I also 
find it very helpful to visit my “clients”’, and see what 
they are doing.’ I have been able to provide them with 
a tool that greatly simplifies their operation (often 
times, it was an existing tool that needed slight modi- 
fication) and I strengthen a person to person relation- 
ship as a result. 


One important lesson, is that your source data is 
often not quite what you need — It may be too early or 
too late (like with HR) requiring some extra tools to 
make it useable, or intended for a different purpose 
and it will need to be adapted and cleaned up (like 
with the departmental tree). You have to work with 
your sources to make it work. The solution may be 
with people and process and not technology. 


Systems and Processes Impacted By This Project 


A number of systems and business processes (see 
Figure 13) has been impacted and improved by this 
project. The most common improvement is the auto- 
matic removal (or at least notification) when people 
leave or their status changes. 


91s this an obvious part of customer service — Yes! Is it of- 
ten overlooked, sadly, Yes. 


For important campus announcements, we have email lists like “All Faculty”, “All 


Staff’, ‘All Students” that are maintained based on current status. Our Postmaster 


Hostmaster 
ID Access Control 


ID Card Admin 


Package Tracking 


PC Store 
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keeps these mappings in place. 

For our host database, there are requirements as to who can “own” or “administrate” 
machines on our campus network. Our Hostmaster makes this determination. 

We have several buildings that are open to the campus “community” via card readers. 
This enables our Access control staff to control that definition of community. 

There are two controls here, one to indicate if that person’s information is maintained 
in Banner (if not, the ID desk can make the updates directly), and also if that person is 
eligible to have a dependent. 

Our campus mail room wants an address feed, but not of everything. They pick and 
choose. 

Our Campus Computer store is limited in who they can sell to (due to contracts with 
vendors). This helps them identify who is ok, and who is eligible to charge their 
purchases to a student account. 


Figure 12: Current auxiliary streams in use. 
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A Note on Table Definitions 


There are a number of Oracle table definitions 
included in this paper. In order to save space and assist 
with formatting, I did not include a number of col- 
umns that are in most of the Simon tables. A recurring 
set of columns that we see in many of the Simon 
tables are When_Inserted, When_Updated and When_ 
Marked_For_Delete columns. These are filled with an 
ever growing sequence, and are used to identify 
records that have changed since some previous point in 
time, and to propagate those changes to other tables and 
systems [4]. Many tables also have a Clerk or Clerk_Id 
field to indicate who touched that record last, and some 
also have an Activity_Date column to indicate in a more 
human form, of when that change took place. I have 
also removed references to those columns in some of 
the source code examples included here. The full defi- 
nitions of both table and source code can be obtained 
via the Web. See the following section for details. 


Futures 


We continue to identify new groups and cate- 
gories of people that need status. Adding these new 
types has become pretty trivial, with the tools and 
techniques being quickly adaptable to new situations. 
One offshoot of this project deals with physical access 
control. We mapped the people status auxiliary values 
into a more general demographic mapping module 
(along with departmental affiliation, course registra- 
tion, etc.) into consumers of group information such as 
the access control system, and windows protection 
groups. Now when a student registers for a particular 
class, not only do they get access (via their ID card), to 
the room with the lab equipment, but access (via their 
computer account) to restricted course files and direc- 
tories on Windows and AFS file servers. There might 
even be another paper on this topic for next year. 


References and Availability 


This is not a comprehensive, stand alone, product 
that we can package up easily for distribution. Despite 
efforts over the years to move our business rules out 
of the code, and into data table and views, there is a 
lot of site specific stuff in here. That being said, I feel 
that there is a lot here that can be used by other sites. 
You will have to do some code development of your 
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own to address your own site’s specific issues and 
requirements. Some of the existing packages we have 
developed may serve as models for your own work. 
The web tools we have developed use a pretty much 
standard Oracle web environment with our own cus- 
tom front end to handle the authentication. 


All of the PL/SQL source code for the Simon sys- 
tem as well as the full table and view descriptions are 
available via the web at http://www.rpi.edu/campus/ 
rpi/simon/misc/Tables/simon.Index.html and http://www. 
rpi.edu/campus/rpi/simon/misc/Tables/SIS-index.html . 
If you ask nicely, I will try to answer questions and 
might be able to dig out some of the C and JAVA code 
that makes up other parts of the system. 
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Library Circulation and Patron database — drives borrowing limits, aids with contacting 


via address feeds. 
Access Control 
community members. 


Automatic revocation of access when a person leaves, automatic access for 


Computer Accounts | Automatic creation and expiration. 


Human Resources 


HR controls when a person will be issued access, email, etc. — ensuring 


compliance with employment rules. 


ID Card Office Single point of contact for guests, identification of departmental contacts, 
refinement and documentation of policies and procedures. 
Directory Status drives inclusion in the online and printed directories. 


Figure 13: Systems and business processes impacted by this system. 
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Appendix A: Selected People _Status_Types 
ID Card Status Source Key Key Nkey 

RNK Status Category Table 1 2 1 Description 

750 Employee Emp Employees A E% Exempt Employees 

750 Employee Emp Employees A X% Executives 

750 Employee Emp Employees A N% Non Exempt employees 

750 Faculty Emp Employees A F% Faculty 

740 Employee NewEmp Employees Staff Newly hired staff 

740 Hartford Emp HartEmp HartfordRawDir Hartford Employees 

740 Faculty NewEmp Employees Faculty Newly hired faculty 

730 ROTC Staff N/A id_people CURRENT 91397008 US Military personal assigned to the 
ROTC detachments in a support (non 
teaching) assignment. 

730 ROTC Faculty N/A id_people CURRENT 91397007 US Military personal assigned to a ROTC 
detachment who will be teaching ROTC 
and other courses. 

730 OnLeave Employee Emp Employees F On Leave with full benefits 

730 Employee (PT) Emp Employees P On Leave w/ Partial Pay and Benefits 

650 Visiting Researcher N/A id_people CURRENT 91399849 A person doing research (but not being 
paid by RPI) 

650 Research Professor N/A id_people CURRENT 91399850 Someone who is teaching, but is not paid 
by RPI. Will have a memo from the 
Provost’s office or the Dean’s office. 

640 Undergrad BStu Banner_students T A Undergrad, in Architecture at Troy 

640 Graduate BStu banner_students T S Graduate, in Science at Troy 

640 Graduate BStu banner_students T E Graduate, in Engineering at Troy 

640 PDE Grad DStu banner_students GR Graduate Student with PDE 

640  HartGrad HStu banner_students GR AS Graduate Student at Hartford 

640 Undergrad BStu banner_students T E Undergrad, in Engineering at Troy 

640 Undergrad BStu banner_students T Ny Undergrad, in Science at Troy 

640 Graduate BStu banner_students T A Graduate, in Architecture at Troy 

560 Incubator Incubatr id_people CURRENT 91068656 A person affiliated with a company in the 
incubator center. 

550  Dependent/Spouse Dependnt id_people CURRENT 91067788 Dependent/Spouse of existing RPI person 

550 Personal Services N/A id_people CURRENT 91449917 A personal aid, generally for health care 
of a disabled member of the Rensselaer 
Community. 

550 Family N/A id_people CURRENT 91449912 A spouse of a member of the RPI 
community 

550 Family N/A id_people CURRENT 91449913 A domestic partner of a member of the 
RPI Community. 

550 Family N/A id_people CURRENT 91449911 A dependent, spouse or domestic partner 

550 Temp Employee TempEmp id_people CURRENT 91318569 A temporary employee, not on payroll — 
JJ Young, Manpower, etc. 

500 Emeritus Faculty DirAux Dir_Aux_Ent MR6190 9 School of Engineering Emeritus Faculty 

500 Emeritus Faculty DirAux  Dir_Aux_Ent MR6210 9 Emeritus Faculty from the School of 
Science 

500 Emeritus Faculty DirAux Dir_Aux_Ent MR6200 9 Emeritus Faculty in Architecture 

300 Vendor Vendor — id_people CURRENT 91067997 A vendor 

300 Retiree Retiree —_id_people CURRENT 91080042 A retiree 

300 Vendor N/A id_people CURRENT 91398702 A person affiliated with an on campus 
vendor 

300 Vendor N/A id_people CURRENT 91402506 A vendor working with a specific 
department on campus. 

300 Vendor N/A id_people CURRENT 91393994 Somone affiliated with an off campus 
vendor 

300. Tech Park TechPark id_people CURRENT 91080046 An employee at a tech park company 

260 RUClub Member N/A id_people CURRENT 91405098 A member of a Rensselaer Union 
sponsored club or organization. 

250 Special Programs Spec Prg__id_people CURRENT 91121096 Special Program 

200 Special Access Spee Acc id_people CURRENT 91068269 Special Access Card 

200 Conference Card ConfCard id_people CURRENT 91114523 A Generic Conference ID card. 

150 Residence Hall ResLife  banner_students A person with an active housing contract. 

Occupant May not be a student nor employee. 
55. __ Retired Faculty DirAux Dir_Aux_Ent S6160 9 A retired faculty member. This status is 
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maintained by the Provost’s office. 
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